Imagine for a moment that, by some quirk of the universe, you are sharing your workspace with a time traveller. Specifically, yourself from 1 year in the future. How will you react to your new co-worker?
That imagination makes one to ponder for a pretty good time :)
Mamba installs these packages in only a third of the time that Conda does. Much of that is due to less CPU usage, but even network downloads seem to be little faster; Mamba uses parallel downloads to speed them up.
Mamba is a lot faster than Conda
So which should you use, pip or Conda? For general Python computing, pip and PyPI are usually fine, and the surrounding tooling tends to be better. For data science or scientific computing, however, Conda’s ability to package third-party libraries, and the centralized infrastructure provided by Conda-Forge, means setup of complex packages will often be easier.
From my experience, I would use Mambaforge or pyenv and Poetry.
My first recommendation would be fiction. Reading fiction is important to understand the cross-sectional variation in humanity, to understand how difficult generalisations can be, to just get a sense of how different social pieces fit together, and to get a sense of different historical eras – and plus, reading fiction is often just plain flat-out fun.
Why reading fiction is important
Quicker And More Useful Communication
Great example of asking precise questions
The different barcode types can be categorized as one-dimensional (1D) and two-dimensional (2D). While one-dimensional barcodes consist only of lines and are usually scanned horizontally with laser scanners, two-dimensional codes make use of vertical space as well and can only be scanned with devices that contain a camera, such as imager scanners or smartphones.
Aparently, there a lot of barcode types
# "func" called 3 times result = [func(x), func(x)**2, func(x)**3] # Reuse result of "func" without splitting the code into multiple lines result = [y := func(x), y**2, y**3]
Smart example of using the walrus operator :=
First of all, the map does a much better job at preserving the relative size and area of land and water masses, while reducing shape distortion. It is also designed to avoid dead ends, allowing the spherical nature of the world to be visualised by simply expanding the map in any direction.
Authagraph map - granted, this map is still not perfect (it's still not a globe) but remains one of the best attempts yet at representing the world in flat, two dimensions.
Most web servers TCP slow start algorithm starts by sending 10 TCP packets. The maximum size of a TCP packet is 1500 bytes.
10 * 1460 bytes = 14 kb.
If your website is under 14 kb, it can load around 612 ms faster than 15 kb.
Second-order thinking is more deliberate. It is thinking in terms of interactions and time, understanding that despite our intentions our interventions often cause harm. Second order thinkers ask themselves the question “And then what?” This means thinking about the consequences of repeatedly eating a chocolate bar when you are hungry and using that to inform your decision. If you do this you’re more likely to eat something healthy.
Second-order thinking
Mob requires good communication skills. There is no space for passive-aggressiveness; or arrogance. If You want to show You are better than Your colleagues, You aren’t a candidate for Mob.
Main requirement of effective mob programming
Even though Chrome, Firefox, and Edge browsers all store passwords in encrypted databases, by default all three products intentionally leave the associated encryption keys completely unprotected in predictable locations.
That's why one should use an external app to store passwords, instead of leaving them in a browser
Blog posts with images get 2.3x more engagement.
Preview: quick before and after
Check out the Preview section to see how much better the blog post images are when generated by DALL·E 2 for $45
If I’m not able to write down any specific experiments, I know the book, video, podcast, article, or thread is just fluff and I should stop reading, listening, watching as soon as possible. This is an incredibly useful tool. Many authors and speakers are extremely skilled at creating the illusion of powerful insights through engaging stories, fortune cookie wisdom, and the use of sophisticated language.
Major tip for consuming non-fiction
Instead, they keep a Thing Table and a Data Table. Everything in Reddit is a Thing: users, links, comments, subreddits, awards, etc. Things keep common attribute like up/down votes, a type, and creation date. The Data table has three columns: thing id, key, value. There’s a row for every attribute. There’s a row for title, url, author, spam votes, etc. When they add new features they didn’t have to worry about the database anymore. They didn’t have to add new tables for new things or worry about upgrades. Easier for development, deployment, maintenance.
Reddit uses only 2 tables, with the cost of not being able to use cool relational features
Schema updates are very slow when you get bigger. Adding a column to 10 million rows takes locks and doesn’t work. They used replication for backup and for scaling. Schema updates and maintaining replication is a pain.
Schema updates and replications are not easy to handle
To Uncover a Deepfake Video Call, Ask the Caller to Turn Sideways
When I find a "get X free" button on a website that then asks for my email address, I like to search for the email of the company behind the website (sometimes it's on the legal page, or the privacy policy page) and I submit their email. I also make sure to check the "sign me up for the newsletter" box, to make sure the spammers get at least one of their messages. I don't really know why I do this, it seemed funny a few months ago when I started and now I do it out of habit. I now keep a list of emails from these spam sites, and subscribe them all to the various newsletters I find if I have 5 minutes.
A little trick to spam the spammers :D
Sit to do computer work. Sit using a height-adjustable, downward titling keyboard tray for the best work posture, then every 20 minutes stand for 8 minutes AND MOVE for 2 minutes. The absolute time isn’t critical but about every 20-30 minutes take a posture break and stand and move for a couple of minutes. Simply standing is insufficient. Movement is important to get blood circulation through the muscles. And movement is FREE! Research shows that you don’t need to do vigorous exercise (e.g. jumping jacks) to get the benefits, just walking around is sufficient. So build in a pattern of creating greater movement variety in the workplace (e.g. walk to a printer, water fountain, stand for a meeting, take the stairs, walk around the floor, park a bit further away from the building each day).
Bottom line: don't just sit or stand at work, but simply change your position often
9 evidence-based common denominators among the world’s centenarians that are believed to slow this aging process.
List of 9 habits that will make you live longer (see text below)
when you use python -m pip with python being the specific interpreter you want to use, all of the above ambiguity is gone. If I say python3.8 -m pip then I know pip will be using and installing for my Python 3.8 interpreter (same goes for if I had said python3.7).
It's better to use python -m pip over pip / pip3 to be sure for which Python version we're installing the dependencies.
However, it's not necessary when using environments.
And if you're on Windows there is an added benefit to using python -m pip as it lets pip update itself. Basically because pip.exe is considered running when you do pip install --upgrade pip, Windows won't let you overwrite pip.exe. But if you do python -m pip install --upgrade pip you avoid that issue as it's python.exe that's running, not pip.exe.
If you would like to update pip on Windows, use python -m pip install --upgrade pip
The only thing that works is to remove yourself from the environment that is causing burnout, and then taking the time off to recover.
The only way to prevent burnout
When you see a job post mentioning REST or a company discussing REST Guidelines they will rarely mention either hypertext or hypermedia: they will instead mention JSON, GraphQL(!) and the like.
It’s time to say goodbye to distutils package and switch to setuptools.
Use setuptools over distutils
as soon as you switch to Python 3.11, you should get into habit of using import tomllib instead of import tomli
tomlib
It's fine to use print if you're debugging an issue locally, but for any production-ready program that will run without user intervention, proper logging is a must.
In production, use logging instead of print
Finally, if you don’t use either namedtuple nor dataclasses you might want to consider going straight to Pydantic.
Pydantic: https://pydantic-docs.helpmanual.io/
You might be wondering why would you need to replace namedtuple? So, these are some reasons why you should consider switching to dataclasses
There are a number of reasons why to prefer dataclasses over namedtuple
Using zoneinfo however has one caveat - it assumes that there's time zone data available on the system, which is the case on UNIX systems. If your system doesn't have timezone data though, then you should use tzdata package which is a first-party library maintained by the CPython core developers, which contains IANA time zone database.
One caveat of zoneinfo
Until Python 3.9, there wasn’t builtin library for timezone manipulation, so everyone was using pytz, but now we have zoneinfo in standard library, so it's time to switch!
Prefer zoneinfo over pytz from Python 3.9
As per docs, random module should not be used for security purposes. You should use either secrets or os.urandom, but the secrets module is definitely preferable, considering that it's newer and includes some utility/convenience methods for hexadecimal tokens as well as URL safe tokens.
Prefer secrets module over os.urandom
pathlib has however many advantages over old os.path - while os module represents paths in raw string format, pathlib uses object-oriented style, which makes it more readable and natural to write
pathlib is the newer os.path with an inclusion of glob.glob
Mappings: https://docs.python.org/3/library/pathlib.html#correspondence-to-tools-in-the-os-module
More about pathlib: https://treyhunner.com/2018/12/why-you-should-be-using-pathlib/
If you need to store duplicates, go for List or Tuple.For List vs. Tuple, if you do not intend to mutate, go for Tuple.If you do not need to store duplicates, always go for Set or Dictionary. Hash maps are significantly faster when it comes to determining if an object is present in the Set (e.g. x in set_or_dict).
Python list vs tuple vs set
How does PostgreSQL access a column? It will fetch the row and then dissect this tuple to calculate the position of the desired column inside the row. So if we want to access column #1000 it means that we have to figure out how long those first 999 columns before our chosen one really are. This can be quite complex. For integer we simply have to add 4, but in case of varchar, the operation turns into something really expensive.
Column order matters in PostgreSQL. The time to read 1000th column can be pretty large, especially if there are varchar columns before that.
WiFi QR code is simply a text QR code with a special format as follows:WIFI:S:<SSID>;T:<WEP|WPA|blank>;P:<PASSWORD>;H:<true|false|blank>;;The S sets the SSID of the network, T defines the security in use, P is the password and H whether the network is hidden or not.
WiFi QR code format
If your laptop is extremely old then I would recommend Puppy Linux.If your laptop isn’t very old but doesn’t perform very well I would recommend AntiX.If your laptop is a little old but still can’t handle Windows 7/10 very well I would recommend Lubuntu.
3 OS recommendations for old laptops: 1. Puppy Linux 2. AntiX 3. Lubuntu
In the functions.php of your WordPress theme add:
Same solution but in PHP
php
function hints() {
header("link: </wp-content/themes/phpied2/style.css>; rel=preload, </wp-includes/css/dist/block-library/style.min.css?ver=5.4.1>; rel=preload");
}
add_action('send_headers', 'hints');
We can. I cannot because the shared Dreamhost hosting doesn't let me edit Apache config beyond .htaccess. But if you have access to the server or virtual host config, see here. Basically all you need to do in addition to what we did in .htaccess is to add: H2EarlyHints on
Even better solution with:
H2EarlyHints on
What I describe here is a one-off thing for demonstration. As you can probably guess by the "5.4.1", when you upgrade WordPress or change themes, you might need to change the URLs of the preloaded CSS in the .htaccess.
Currently there may be no plugin to do it automatically
And immediately after it, the 2 CSS downloads begin. What we want to do is move the CSS downloads to the left, so all rendering starts (and finishes!) sooner. So all you do it take the URLs of these two files and add them to .htaccess with H2PushResource in front. For me that means the URL to my custom theme's CSS /wp-content/themes/phpied2/style.css as well as some WordPress CSS stuff. While I was there I also added a JavaScript file which is loaded later. Why now start early? So the end result is:
WordPress tip to start loading some CSS and JS files earlier.
Sample code to add to .htaccess:
H2PushResource /wp-content/themes/phpied2/style.css
H2PushResource /wp-includes/css/dist/block-library/style.min.css?ver=5.4.1
H2PushResource /wp-includes/js/wp-emoji-release.min.js?ver=5.4.1
If you’ve ever felt that you can’t focus on a task when you’re hungry — or that all you can think about is food — the neural evidence backs you up. Work from a few years ago confirmed that short-term hunger can change neural processing and bias our attention in ways that may help us find food faster.
They are:doers: self-motivated or intrinsically driven to achieve and learndrivers: proactive and decisive; don't wait for orderspromoters: passionate and excited to share what they create for their own benefit and the benefit of others
3 abilities of great async workers
Review: a formal assessment or examination of something with the possibility or intention of instituting change if necessary.
(comment with good examples of code review comments)
On top of that, there’s one thing you can do to extend your life. By studying the philosophies of those who came before you, you absorb their experiences. Every philosophy book you read, you’re adding the author’s lifespan to yours. There’s no better way to spend your time than studying philosophy.13
What I was trying to say before was just because someone’s always busy, and lives to an old age, doesn’t mean they’ve lived long. They’ve just existed long.
I’m not saying you should lay down on the beach all day. I’m saying you should find something that’s enjoyable to you, and valuable for the world.10 You should live your life intentionally, instead of having your time stolen from you little by little.
Even with all that, he was looking forward to the day that he could step down and retire from it all. The man with all the power in the world was happiest when he thought about the day he could let go of all the power.8 How foolish is it to spend your life chasing fame, riches, and power, while being unhappy the entire time, even after you achieve it?
The most surprising thing is that you wouldn’t let anyone steal your property, but you consistently let people steal your time, which is infinitely more valuable.2
sudo killall coreaudiod
Disable flashing screen on specific sounds, even while pressing backspace in a Python terminal and hitting the left end.
This is a neat Docker trick for those who have an ARM development machine (Apple M1), but sometimes need to build x86/amd64 images locally to push up to a registry.
Since Apple M1 is based on the ARM architecture, it is still possible to build images based on Linux x86/amd64 architecture using docker buildx:
docker buildx build -f Dockerfile --platform linux/amd64 .
However, building such images can be really slow, so we can create a builder profile (see the paragraphs below / my other annotation to this article).
So, we can create this builder on our local machine. The nice part about this creation is that it is idempotent, so you can run this command many times without changing the result. All we have to do is to create a builder profile and in this case I have named it amd64_builder.
Example of creating a Docker buildx builder profile on the Apple M1 machine. This will allow for around 10x faster builds on the amd64 architecture pushed to a registry, than on the amd64 emulation on the M1 chip.
sum(1 to n) = n(n + 1) / 2
Python 3.11 is up to 10-60% faster than Python 3.10. On average, we measured a 1.25x speedup on the standard benchmark suite. See Faster CPython for details.
On the speed of Python 3.11
But to my knowledge, this is the first time anyone has collected public info about Bigco dev environments in one place
Examples of big companies moving to development in the cloud
Git also has a built-in command (maintenance) to optimize a repository’s data, speeding up commands and reducing disk space. This isn’t enabled by default, so we register it with a schedule for daily and hourly routines.
git maintenance
On a new clone of the Canva monorepo, git status takes 10 seconds on average while git fetch can take anywhere from 15 seconds to minutes due to the number of changes merged by engineers.
Over the last 10 years, the code base has grown from a few thousand lines to just under 60 million lines of code in 2022. Every week, hundreds of engineers work across half a million files generating close to a million lines of change (including generated files), tens of thousands of commits, and merging thousands of pull requests.
The perpetrator in question was completing an internship and committed code into the Windows 3.1 code base that was a little prank for the test team: Under a very specific error condition, it changed the index finger pointer to a middle finger.
Funny/rude prank in Windows 3.1
23.0G com.txt # 23 gigs uncompressed
23 GB txt file <--- list of all the existing .com domains
What makes you unique is not your specific attributes. It’s your specific ties to the network around you.
Becoming disentangled from your web of mutual commitments, shared history, and collective responsibility is to be rendered into a transaction, a slave.
Real friendships don’t form via shared interests. They form via shared context.
In a Staging workflow, releases are slower because of more steps, and bigger because of batching.
For Staging to be useful, it has to catch a special kind of issues that 1) would happen in production, but 2) wouldn’t happen on a developer's laptop.What are these? They might be problems with data migrations, database load and queries, and other infra-related problems.
How "Staging" environment can be useful
Another disadvantage of managed platforms is that they are inflexible and slow to change. They might provide 80% of the functionality we require, but it is often the case that the missing 20% provides functionality that is mission critical for machine learning projects. The closed design and architecture of managed platforms makes it difficult to make even the most trivial changes. To compensate for this lack of flexibility, we often have to design custom, inefficient and hard-to-maintain mechanisms that add technical debt to the project.
Main disadvantage of managed MLOps platforms
As you've probably already guessed, we've decided to replace the current Web IDE with one built on top of VS Code. In the coming milestones, we will build out custom support for the features not already available in the VS Code core, and validate that the workflows you already depend on in the Web IDE are handled in the new experience. We're working with the team that builds our amazing GitLab Workflow extension for VS Code to make it available in the browser so we can bundle it in the Web IDE, and bring all those great features along for the ride. That includes bringing merge request comments into the Web IDE for the first time ever!
GitLab is planning to onboard VS Code web IDE
A normal Makefile for building projects with Docker would look more or less like this:
Sample of a Makefile for building and tagging Docker images
One of the main benefits from tagging your container images with the corresponding commit hash is that it's easy to trace back who a specific point in time, know how the application looked and behaved like for that specifc point in history and most importantly, blame the people that broke the application ;)
Why tagging Docker images with SHA is useful
Software Bill Of Materials (SBOM) is analogous to a packing list for a shipment; it’s all the components that make up the software, or were used to build it. For container images, this includes the operating system packages that are installed (e.g.: ca-certificates) along with language specific packages that the software depends on (e.g.: log4j). The SBOM could include only some of this information or even more details, like the versions of components and where they came from.
Software Bill Of Materials (SBOM)
Included in Docker Desktop 4.7.0 is a new, experimental docker sbom CLI command that displays the SBOM (Software Bill Of Materials) of any Docker image.
New docker sbom CLI command
With every generation of computing comes a dominant new software or hardware stack that sweeps away the competition and catapults a fledgling technology into the mainstream.I call it the Canonical Stack (CS).Think the WinTel dynasty in the 80s and 90s, with Microsoft on 95% of all PCs with “Intel inside.” Think LAMP and MEAN stack. Think Amazon’s S3 becoming a near universal API for storage. Think of Kubernetes and Docker for cloud orchestration.
Explanation of Canonical Stack (CS)
By offering to have a real-time technical discussion, you are reframing the code review process away from an 'instructor correcting student' dynamic and into a 'colleagues working together' mindset.
if you are on the job market looking for a team that cares more about being agile than going through the motions to look agile, ask these questions
Level 5: Stop the line. The highest level of code review comments. Borrowing the term from Toyota's manufacturing process this is when the code reviewer noticed something in the PR that signals a major defect.
Stop the line - 5th type of MR comments
Level 4: Infringement. This is where things get more serious, note that infringement means rules were broken. In this context, rules can mean a number of things, from the more obvious feature spec and framework rules to things like style guides and coding principles.
Infringement - 4th type of MR comments
Level 3: Suggestions. These can also be thought of as recommendations and alternatives.
Suggestions - 3rd type of MR comments
Level 2: Nitpicks. Usually, comments about grammar errors and minor stylistic issues/typos go there. The solution to the nitpicks is usually very obvious and if the solution is opinionated, the opinion is not strongly held. Naming a method foobarGenerator vs foobarFactory goes in here, and nitpick comments often start with Nitpick:.
Nitpicks - 2nd type of MR comments
Level 1: Clarifications. I also think of these as sanity checks, I may even start the comment with something like I may be completely off the base here... or Just to sanity check...
Clarifications - 1st type of MR comments
Each developer on average wastes 30 minutes before and after the meeting to context switch and the time is otherwise non-value adding. (See this study for the cost of context switching).
Overall, if speed is your primary concern and you’re on a budget, then Circle CI is the clear choice. If you’re not looking to run a ton of builds each month and your code is already in Github, then Github Actions can offer similar performance with the added convenience of having everything under one service. Even though we liked Travis better, our main criteria was value, and since you can’t use Travis for free after the first month, GitLab was able to grab the third slot, despite it being weaker in almost every other category.
4 CI free tier comparison: * Quality of Documentation * Compute Power * Available Disk Space * Free Build Minutes * Speed and Performance
Somebody once described balance to me as three buckets filled with water. One for career, a second for physical health, and a third for social and family life. At any point, one bucket might be running low. But as long as the overall water level is high enough, things should be fine.
Balance represented by 3 buckets
html[theme='dark-mode'] { filter: invert(1) hue-rotate(180deg); }
Ultimate CSS to turn on dark mode
Pyenv works by adding a special directory called shims in front of your PATH environment variable
How pyenv works
If you are on Linux, you can simply download it from GitHub but the most convenient way is to use the pyenv-installer that is a simple script that will install it automatically on your distro, whatever it is, in the easiest possible way.
Installing pyenv on Linux
As of today, the Docker Engine is to be intended as an open source software for Linux, while Docker Desktop is to be intended as the freemium product of the Docker, Inc. company for Mac and Windows platforms. From Docker's product page: "Docker Desktop includes Docker Engine, Docker CLI client, Docker Build/BuildKit, Docker Compose, Docker Content Trust, Kubernetes, Docker Scan, and Credential Helper".
About Docker Engine and Docker Desktop
The diagram below tries to summarise the situation as of today, and most importantly to clarify the relationships between the various moving parts.
Containers (the backend):
If you want your voice to be heard (and also improve the usability of your text) you have to design your document for “skim-ability”. You do this by providing anchor points that allow the user to gauge the content without actually reading it. You want the outline and key arguments to keep standing out in the final version of the document.
This is why I like using bullet points
When writing an article, I generally visualize a concrete person as representative of the audience, that I am directing this text towards.
I also tend to write to my old self
Writing is generally a great way to learn, but one has to realize that you are doing it. Learning is a slow process and requires patience. It is not helped much by agonizing in front of a screen, trying to squeeze out another sentence. Doing more research on the topic by reading a book, blog or paper and taking notes may be a better time investment.
Writing is a great learning method
The realization that you don’t have the complete message in your head, will often only become apparent while writing. This surfaces as inability to find a good punch-line or to express yourself clearly. In fact, writing is a great test to see if you have a good understanding of a topic, and have a firm grasp on the vocabulary of the domain.
an asshole test. You see who turns into an asshole under pressure and they don't make it to the next round".
Asshole test: 60 minutes, 8 people, no right answer, but checking who turns into an asshole under pressure
A 20-year age difference (for example, from 20 to 40, or from 30 to 50 years old) will, on average, correspond to reading 30 WPM slower, meaning that a 50-year old user will need about 11% more time than a 30-year old user to read the same text.
Users’ age had a strong impact on their reading speed, which dropped by 1.5 WPM for each year of age.
Stop using TODO for everything in your comments that requires you to do something later.
Possible alternatives of TODO: * FIXME - something is broken * HACK/OPTIMIZE - the code is suboptimal and should be refactored * BUG - there is a bug in the code * CHECKME/REVIEW - the code needs to be reviewed * DOCME - the code needs to be documented (either in codebase or external documentation) * TESTME - the specified code needs to be tested or that tests need to be written for that selection of code
Create the new empty table Write to both old and new table Copy data (in chunks) from old to new Validate consistency Switch reads to new table Stop writes to the old table Cleanup old table
7 steps required while migrating to a new table
Without accounting for what we install or add inside, the base python:3.8.6-buster weighs 882MB vs 113MB for the slim version. Of course it's at the expense of many tools such as build toolchains3 but you probably don't need them in your production image.4 Your ops teams should be happier with these lighter images: less attack surface, less code that can break, less transfer time, less disk space used, ... And our Dockerfile is still readable so it should be easy to maintain.
See sample Dockerfile above this annotation (below there is a version tweaked even further)
scratch is a special empty image with no operating system.
FROM scratch
Overall, having spent a significant amount of time building this project, scaling it up to the size it’s at now, as well as analysing the data, the main conclusion is that it is not worth building your own solution, and investing this much time. When I first started building this project 3 years ago, I expected to learn way more surprising and interesting facts. There were some, and it’s super interesting to look through those graphs, however retrospectively, it did not justify the hundreds of hours I invested in this project.I’ll likely continue tracking my mood, as well as a few other key metrics, however will significantly reduce the amount of time I invest in it.
Words of the author of https://krausefx.com//blog/how-i-put-my-whole-life-into-a-single-database
It seems as if excessive personal data tracking is not worth it
45% less time spent in video & audio calls that day
I can relate with the author. Spending time in a high number of video/audio calls can drastically decrease my mood
A properly formed Git commit subject line should always be able to complete the following sentence:If applied, this commit will your subject line hereFor example:If applied, this commit will refactor subsystem X for readability
An example how to always aim for imperative commits
Git itself uses the imperative whenever it creates a commit on your behalf.For example, the default message created when using git merge reads:Merge branch 'myfeature'
Using imperative mood in subject line of git commits
Commit messages with bodies are not so easy to write with the -m option. You’re better off writing the message in a proper text editor.
I've tested it on Windows, and in PowerShell or Git Bash it is as simple as:
```console git commit -m "Subject line<ENTER>
body line 1 body line 2"<ENTER> ```
However, it does not work in CMD.exe (pressing [ENTER] will not move to the next line)
Firstly, not every commit requires both a subject and a body. Sometimes a single line is fine, especially when the change is so simple that no further context is necessary.
Not every commit requires a body part
Summarize changes in around 50 characters or less More detailed explanatory text, if necessary. Wrap it to about 72 characters or so. In some contexts, the first line is treated as the subject of the commit and the rest of the text as the body. The blank line separating the summary from the body is critical (unless you omit the body entirely); various tools like `log`, `shortlog` and `rebase` can get confused if you run the two together. Explain the problem that this commit is solving. Focus on why you are making this change as opposed to how (the code explains that). Are there side effects or other unintuitive consequences of this change? Here's the place to explain them. Further paragraphs come after blank lines. - Bullet points are okay, too - Typically a hyphen or asterisk is used for the bullet, preceded by a single space, with blank lines in between, but conventions vary here If you use an issue tracker, put references to them at the bottom, like this: Resolves: #123 See also: #456, #789
Example of a great commit message
Separate subject from body with a blank lineLimit the subject line to 50 charactersCapitalize the subject lineDo not end the subject line with a periodUse the imperative mood in the subject lineWrap the body at 72 charactersUse the body to explain what and why vs. how
7 rules of great commit messages
What may be a hassle at first soon becomes habit, and eventually a source of pride and productivity for all involved.
Commit messages can do exactly that and as a result, a commit message shows whether a developer is a good collaborator.
A diff will tell you what changed, but only the commit message can properly tell you why.
Commit messages are important
Look at Spring Boot, or any repository managed by Tim Pope.
2 great examples of clean commits: * https://github.com/spring-projects/spring-boot/commits/master * https://github.com/tpope/vim-pathogen/commits/master
To access your Linux files in Windows, open the Ubuntu terminal and type explorer.exe . (include the punctuation mark). This will open the linux directory in Windows Explorer, with the WSL prefix “\wsl$\Ubuntu-18.04\home\your-username”.Now, you’ll notice that Windows treats your Linux environment as a second network.
The eagle-eyed among you may notice that there isn't a .git directory in the app-example-2 working tree. Instead, there's a .git file. This file points to the git directory in the original clone, and it means that all your usual git commands work inside the app-example-2 directory, as well as in the original app-example directory.
The working of git worktree
Anecdotally, I've found IDEs get much less "confused" if you use their built-in support for switching git branches, instead of changing them from the command line and waiting for the IDE to "notice" the changes.
Reading puts candidates at ease compared to writing code. As an interviewer, stress is your enemy because it raises adrenaline which lowers IQ by several points, causing you to miss good candidates. Candidates prefer reading partly because they are relieved to not have to write code, but also because the interviewer can easily adjust the reading questions to accommodate for the candidate’s skill.
Finally, to make our terminal really pretty, we need to customize the prompt. There's lots of options out there for this, but the most popular one seems to be ohmyzsh for Bash and oh-my-posh for PowerShell. I'm not a huge fan of these because in my experience they slow down the terminal to a point which makes me frustrated to use them, and since they are separate solutions for each environment they must be configured separately.
I agree. After using oh-my-posh in almost every Windows console, I have finally decided to make a switch to Starship
We asked them to collect searches they performed in their normal workday, and then evaluate their performance on Google, Bing, Neeva, You.com, Kagi, and DuckDuckGo.Here are more examples of programming queries where Google is lagging behind.
For example, searching for "python throw exception" will show better results in Neeva or You.com than in Google
In the previous version, using the standard library, once the data is loaded we no longer to keep the file open. With this API the file has to stay open because the JSON parser is reading from the file on demand, as we iterate over the records.
For ijson.items(), the peak tracked memory usage was 3.6 MiB for a large JSON, instead of 124.7 MiB as for the standard json.load()
One common solution is streaming parsing, aka lazy parsing, iterative parsing, or chunked processing.
Solution for processing large JSON files in Python
Then, if the string can be represented as ASCII, only one byte of memory is used per character. If the string uses more extended characters, it might end up using as many as 4 bytes per character. We can see how much memory an object needs using sys.getsizeof()
"a" takes less bytes than "❄", which takes less bytes than "💵"
I tried PuTTY, which I’d heard was the good SSH thing on Windows, but it’s… not good, at all. PowerShell does come with an SSH client, so once you have this working with a reasonable terminal, you can use SSH as normal.
SSH that comes with Windows/PowerShell over PuTTY
I sometimes wondered why the VS Code team put so much effort into the built-in terminal inside the editor. I tried it once on Linux and never touched it again, because the terminal window I had right next to my editor was just massively better in every way. Having used Windows terminals for a while, I now fully understand why it’s there.
VS Code terminal is not as efficient on Linux
Scoop seems to be the best of the bunch, so far?
Scoop seems to be the best package manager
They just automate the process of going to the website, downloading an installer and then running it - which is slightly better than doing it yourself.
Windows package managers are unlike Linux ones
It’s got less useful stuff in it than most Linux distro “app stores” and is utterly miniscule compared to the Debian repositories, which have ~60,000 packages in, or Arch’s AUR, with 73,000 (these counts include the whole Linux OS, though, with is installed using the same package manager).
Windows Store
if you want to add persistent environment variables to your currently running shell, you should put a setx command in your $profile file and then reload it: . $profile - or maybe run myvar="value" && setx %myvar% "value", or something similar.
Storing persisent environment variables on Windows
C:\Users\DuncanLock\Documents\PowerShell\Microsoft.PowerShell_profile.ps1~/.bashrc
:(
PowerShell does have a ~ alias for your home folder, and cd ~ works!
You can get Windows versions of most of the standard *nix userland utils, which seem to work OK with PowerShell
scoop install coreutils
Desktop Linux is often criticized for this, but Windows is much worse, somehow! It’s really inconsistent. Half of it is “new” UI and half of it is old Win32/GDI type UI - just as bad as KDE/GTK - except worse, because you can’t configure them to use the same theme. Also, when you install a Linux distribution, it’ll start off either all KDE or all GTK, or whatever - but with Windows you’re stuck with a random mix of both right from the start.
Windows is a mess...
Most companies are not prepared to pay for a staging environment identical to production
Keeping staging environment has its cost
The Code Review Pyramid
With personal stability (job, hobbies, network, etc.) and age you care less and less what people think and put less emphasis on what people think about you. - At the same time, you judge people less and less because, what does your judgment really matter? And when you judge people less you judge yourself less. - A lot of awkwardness is a function of trying hard to fit a social shape and failing. When you let go of most social expectations of yourself and others, in my experience mostly because of age and stability, you lose the awkwardness.
Lists promise value quickly A normal essay promises a payoff only when you finish it. But lists promise at least some value almost immediately.
Let’s look at a recent paper by Xia, Bao, Lo, Xing, Hassan, & Li entitled Measuring Program Comprehension: A Large-Scale Field Study with Professionals and published in IEEE Transactions on Software Engineering, 44, 951-976, 2018. This paper is quite interesting in that it describes in great details how the figures are obtained. And it says that Comprehension took on average ~58%.
Developers spend most of their time figuring the system out
# Input Input: 123, Output: Input: 121, Output: Input: 111, Output: Input: 123454321, Output: Input 123123, Output: # Instruction Output true if input is a palindrome # Output Input: 123, Output: false Input: 121, Output: true Input: 111, Output: true Input: 123454321, Output: true Input 123123, Output: false
Example of using GPT-3 for programming
Using named arguments is nice for languages that support it, but this is not always a possibility. Even in Python, where time.sleep is defined with a single argument named secs, we can’t call sleep(secs=300) due to implementation reasons. In that case, we can give the value a name instead.Instead of this:time.sleep(300)Do this:sleep_seconds = 300 time.sleep(sleep_seconds)Now the code is unambiguous, and readable without having to consult the documentation.
Putting units in variable names
defaults write NSGlobalDomain WebKitDeveloperExtras -bool true defaults write -g WebKitDeveloperExtras -bool YES
Type these 2 commands to macOS terminal to be able to "Inspect Elements" in a Safari Web Inspector
Core: like an engine in a car. The product is meaningless without it.Necessary: like a car’s spare wheel. It’s rarely used but when needed, its function decides the success of the system.Added value: like a car’s cup-holder. It’s nice to have but the product is perfectly usable without it.Unique Selling Point: the main reason people should buy your product instead of your rivals. For example, your car is the best off-road vehicle.
4 categories of software features
Here is an architecture diagram of the devices I’m currently using
Architecture of recommended Smart Home products (March 2022)
Have you ever built an image only to realize that you actually need it on a user account other than root, requiring you to rebuild the image again in rootless mode? Or have you built an image on one machine but run containers on the image using multiple different machines? Now you need to set up an account on a registry, push the image to the registry, Secure Shell (SSH) to each device you want to run the image on, and then pull the image. The podman image scp command solves both of these annoying scenarios as quickly as they occur.
Podman 4.0 can transfer container images without a registry.
For example: * You can copy a root image to a non-root account:
$ podman image scp root@localhost::IMAGE USER@localhost::
* Or copy an image from one machine to another with this command:
$ podman image scp me@192.168.68.122::IMAGE you@192.168.68.128::
an onion address is a promise and a mechanism to assure that you are taking seriously the needs of the people who use Tor.
Why offer an Onion Address rather than just encourage browsing-over-Tor
As mentioned earlier, PATCH requests should apply partial updates to a resource, whereas PUT replaces an existing resource entirely. It's usually a good idea to design updates around PATCH requests
Prefer PATCH over PUT
Aside from using HTTP status codes that indicate the outcome of the request (success or error), when returning errors, always use a standardized error response that includes more detailed information on what went wrong.
For example: ``` // Request => GET /users/4TL011ax
// Response <= 404 Not Found { "code": "user/not_found", "message": "A user with the ID 4TL011ax could not be found." } ```
https://api.averagecompany.com/v1/health https://api.averagecompany.com/health?api_version=1.0
2 examples of versioning APIs
When dealing with date and time, APIs should always return ISO 8601-formatted strings.
So, to summarize, don't ask "Any Java experts around?", but rather ask "How do I do [problem] with Java and [other relevant info]?"
Quick advice on forming questions
You're asking people to take responsibility. You're questioning people's confidence in their abilities. You're also unnecessarily walling other people out. I often answer questions related to languages or libraries I have never used, because the answers are (in a programmer kind of way) common sense.
Don't ask unspecified questions like "Any Java experts around?"
This isn’t MAIN_PRIORITY, so we aren’t going to do it until at least ESTIMATED_DONE_DATE. Right now our priority is MAIN_PRIORITY because of ONE_SENTENCE_JUSTIFICATION, and this is 100% our shipping focus. I agree this sounds like a really useful feature - once we finish MAIN_PRIORITY, should we consider dropping SECOND_PRIORITY and do it?
Templates for saying NO
for debugging purposes, a good combination is --lf --trace which would start a debug session with pdb at the beginning of the last test that failed:
pytest --lf --trace
pytest -l
Show values of local variables in the output with -l
If you start pytest with --pdb, it will start a pdb debugging session right after an exception is raised in your test. Most of the time this is not particularly useful as you might want to inspect each line of code before the raised exception.
The --pdb option for pytest
pytest --lf
Run the last failed test only with --lf
Run all tests, but run the last failed ones first with --ff
pytest -x
Exiting on the 1st error with -x
pytest --collect-only
Collecting Pytests (not running them)
pytest test_validator.py::test_regular_email_validates
Example of running just one test (test_regular_email_validates) from test_validator.py
Apart from shared fixtures you could place external hooks and plugins or modifiers for the PATH used by pytest to discover tests and implementation code.
Additional things to store in conftest.py
pytest can read its project-specific configuration from one of these files: pytest.ini tox.ini setup.cfg
3 options for configuring pytest
To have the fixture actually be used by one of your test, you simply add the fixture’s name as an argument
Example:
```python import pytest
@pytest.fixture() def database_environment(): setup_database() yield teardown_database()
def test_world(database_environment): assert 1 == 1 ```
But the problem with Poetry is arguably down to the way Docker’s build works: Dockerfiles are essentially glorified shell scripts, and the build system semantic units are files and complete command runs. There is no way in a normal Docker build to access the actually relevant semantic information: in a better build system, you’d only re-install the changed dependencies, not reinstall all dependencies anytime the list changed. Hopefully someday a better build system will eventually replace the Docker default. Until then, it’s square pegs into round holes.
Problem with Poetry/Docker
Third, you can use poetry-dynamic-versioning, a plug-in for Poetry that uses Git tags instead of pyproject.toml to set your application’s version. That way you won’t have to edit pyproject.toml to update the version. This seems appealing until you realize you now need to copy .git into your Docker build, which has its own downsides, like larger images unless you’re using multi-stage builds.
Approach of using poetry-dynamic-versioning plugin
But if you’re doing some sort of continuous deployment process where you’re continuously updating the version field, your Docker builds are going to be slow.
Be careful when updating the version field of pyproject.toml around Docker
VCR.py works primarily via the @vcr decorator. You can import this decorator by writing: import vcr.
How VCR.py works
The VCR.py library records the responses from HTTP requests made within your unit tests. The first time you run your tests using VCR.py is like any previous run. But the after VCR.py has had the chance to run once and record, all subsequent tests are:Fast! No more waiting for slow HTTP requests and responses in your tests.Deterministic. Every test is repeatable since they run off of previously recorded responses.Offline-capable! Every test can now run offline.
VCR.py library to speed up Python HTTP tests
75% of people in the U.S. never tweet.On an average weeknight in January, just 1% of U.S. adults watched primetime Fox News (2.2 million). 0.5% tuned into MSNBC (1.15 million).Nearly three times more Americans (56%) donated to charities during the pandemic than typically give money to politicians and parties (21%).
The ENTRYPOINT specifies a command that will always be executed when the container starts. The CMD specifies arguments that will be fed to the ENTRYPOINT.
Another great comparison of ENTRYPOINT and CMD command
DevOps is an interesting case study for understanding MLOps for a number of reasons: It underscores the long period of transformation required for enterprise adoption.It shows how the movement is comprised of both tooling advances as well as shifts in cultural mindset at organizations. Both must march forward hand-in-hand.It highlights the emerging need for practitioners with cross-functional skills and expertise. Silos be damned.
3 things MLOps can learn from DevOps
MLOps today is in a very messy state with regards to tooling, practices, and standards. However, this is to be expected given that we are still in the early phases of broader enterprise machine learning adoption. As this transformation continues over the coming years, expect the dust to settle while ML-driven value becomes more widespread.
State of MLOps in March 2022
LXC, is a serious contender to virtual machines. So, if you are developing a Linux application or working with servers, and need a real Linux environment, LXC should be your go-to. Docker is a complete solution to distribute applications and is particularly loved by developers. Docker solved the local developer configuration tantrum and became a key component in the CI/CD pipeline because it provides isolation between the workload and reproducible environment.
LXC vs Docker
# line containing 'cake' but not 'at' # same as: grep 'cake' table.txt | grep -v 'at' # with PCRE: grep -P '^(?!.*at).*cake' table.txt $ awk '/cake/ && !/at/' table.txt blue cake mug shirt -7
It should be easier to use awk over bash, especiallly for AND conditions.
For example, for "line containing cake but not at":
* grep: grep 'cake' table.txt | grep -v 'at'
* grep with PCRE: grep -P '^(?!.*at).*cake' table.txt
* awk: awk '/cake/ && !/at/' table.txt
Jeżeli masz dylemat czy użyć CMD, czy ENTRYPOINT jako punkt startowy twojego kontenera, odpowiedz sobie na następujące pytanie.Czy zawsze moje polecenie MUSI się wykonać? Jeśli odpowiedź brzmi tak, użyj ENTRYPOINT. Co więcej, jeśli potrzebujesz przekazać dodatkowe parametry, które mogą być nadpisane podczas uruchomienia kontenera — użyj również instrukcji CMD.
How to simply decide if to use CMD or ENTRYPOINT in a Dockerfile
More backlinks to you and your work: Being the teammate that contributes to the system of knowledge shared shows how much you care about the success of the organization. And, it does it in a way that helps you have more documented and attributable credibility for the value you create within your organization.
Why it is important to write clear documentation in an organization
Writing supplants meetings: When there is good documentation around a meeting (briefs, meeting notes, etc.), meetings can be leaner and more productive because people don’t have to be in the room to know what’s happening. So, only those who are actively contributing to the discussion need to attend.
Without meeting notes and documentation, companies become reliant on unreliable verbal accounts, 1:1 updates, and needing to be in the room to get things done.
Scales everyone’s knowledge: think about how many people you interact with on a given work day. I’m talking about the real human to human kind where you relay your ideas to others. It’s probably in the 5-20 range. If you put those same ideas into a doc or an email, your distribution goes to infinity. Anyone can read it.
Writing can greatly scale up
== and != for string comparison -eq, -ne, -gt, -lt, -le -ge for numerical comparison
Comparison syntax in Bash
> will overwrite the current contents of the file, if the file already exists. If you want to append lines instead, use >>
> - overwrites text
>> - appends text
The syntax for “redirecting” some output to stderr is >&2. > means “pipe stdout into” whatever is on the right, which could be a file, etc., and &2 is a reference to “file descriptor #2” which is stderr.
Using stderr. On the other hand, >&1 is for stdout
single quotes, which don’t expand variables
In Bash, double quotes ("") expand variables, whereas single quotes ('') don't
This only works if you happen to have Bash installed at /bin/bash. Depending on the operating system and distribution of the person running your script, that might not necessarily be true! It’s better to use env, a program that finds an executable on the user’s PATH and runs it.
Shebang tip: instead of ```
use
alternatively, you can replace `bash` with `python`, `ruby`, etc. and later chmod it and run it:
$ chmod +x my-script.sh
$ ./my-script.sh
```
It isn't about note taking, it's about supporting a brain that simply isn't capable of retaining the level of information that we have to deal with. The ultimate note taking device is actually an augmented human brain that has perfect recollection and organisation.
What note taking is about
In Japan, the country where the first emoji sets originated, red is traditionally used to represent increases in the value of a stock. Meanwhile, green is used to represent decreases in stock value.
Reason why chart increasing emoji 📈 is red
the curse of knowledge. It’s a simple but devastating effect: Once we know something, it’s very difficult to imagine not knowing it, or to take the perspective of someone who doesn't.
The curse of knowledge
Adopting Kubernetes-native environments ensures true portability for the hybrid cloud. However, we also need a Kubernetes-native framework to provide the "glue" for applications to seamlessly integrate with Kubernetes and its services. Without application portability, the hybrid cloud is relegated to an environment-only benefit. That framework is Quarkus.
Quarkus framework
Kubernetes-native is a specialization of cloud-native, and not divorced from what cloud native defines. Whereas a cloud-native application is intended for the cloud, a Kubernetes-native application is designed and built for Kubernetes.
Kubernetes-native application
According to Wilder, a cloud-native application is any application that was architected to take full advantage of cloud platforms. These applications: Use cloud platform services. Scale horizontally. Scale automatically, using proactive and reactive actions. Handle node and transient failures without degrading. Feature non-blocking asynchronous communication in a loosely coupled architecture.
Cloud-native applications
two main problems with framing decisions and policies in terms of usefulness: (1) being useful is not always to our own benefit – sometimes, we are being used as a means to someone else’s end, and we end up miserable as a result; and (2) the lenses themselves of usefulness and uselessness can obscure our view of the good life.
2 main problems of usefulness
The script is in batch with some portions of powershell. The base code is fairly simple and most of it came from Googling ".bat transfer files" followed by ".bat how to only transfer certain file types" etc. The trick was making it work with my office, knowing where to scan for new files, knowing where not to scan due to lag (seriously, if you have a folder with 200,000 .txt files that crap will severally slow down your scans. Better to move it manually and then change the script to omit that folder from future searches)
It essentially scans the on-site drive for any new files, generates hash values for them, transfers them to the Cloud, then generates hash values again for fidelity (in court you have to prove digital evidence hasn't been tampered with).
Script to automate an 8 hour job: The firm gets thousands of digital documents, photos, etc on a daily basis. All of this goes on a local drive. My job is to transfer all of these files to the Cloud and then verify their fidelity.
Salesforce has a unique use case where they need to serve 100K-500K models because the Salesforce Einstein product builds models for every customer. Their system serves multiple models in each ML serving framework container. To avoid the noisy neighbor problem and prevent some containers from taking significantly more load than others, they use shuffle sharding [8] to assign models to containers. I won’t go into the details and I recommend watching their excellent presentation in [3].
Case of Salesforce serving 100K-500K ML models with the use of shuffle sharding
Batching predictions can be especially beneficial when running neural networks on GPUs since batching takes better advantage of the hardware.
Barching predictions
Inference Service — provides the serving API. Clients can send requests to different routes to get predictions from different models. The Inference Service unifies serving logic across models and provides easier interaction with other internal services. As a result, data scientists don’t need to take on those concerns. Also, the Inference Service calls out to ML serving containers to obtain model predictions. That way, the Inference Service can focus on I/O-bound operations while the model serving frameworks focus on compute-bound operations. Each set of services can be scaled independently based on their unique performance characteristics.
Responsibilities of Inference Service
Provide a model config file with the model’s input features, the model location, what it needs to run (like a reference to a Docker image), CPU & memory requests, and other relevant information.
Contents of a model config file
what changes when you need to deploy hundreds to thousands of online models? The TLDR: much more automation and standardization.
MLOps focuses deeply on automation and standardization
“Shadow Mode” or “Dark Launch” as Google calls it is a technique where production traffic and data is run through a newly deployed version of a service or machine learning model, without that service or model actually returning the response or prediction to customers/other systems. Instead, the old version of the service or model continues to serve responses or predictions, and the new version’s results are merely captured and stored for analysis.
Shadow mode
you can also mount different FastAPI applications within the FastAPI application. This would mean that every sub-FastAPI application would have its docs, would run independent of other applications, and will handle its path-specific requests. To mount this, simply create a master application and sub-application file. Now, import the app object from the sub-application file to the master application file and pass this object directly to the mount function of the master application object.
It's possible to mount FastAPI applications within a FastAPI application
There are officially 5 types of UUID values, version 1 to 5, but the most common are: time-based (version 1 or version 2) and purely random (version 3). The time-based UUIDs encode the number of 10ns since January 1st, 1970 in 7.5 bytes (60 bits), which is split in a “time-low”-“time-mid”-“time-hi” fashion. The missing 4 bits is the version number used as a prefix to the time-hi field. This yields the 64 bits of the first 3 groups. The last 2 groups are the clock sequence, a value incremented every time the clock is modified and a host unique identifier.
There are 5 types of UUIDs (source):
Type 1: stuffs MAC address+datetime into 128 bits
Type 3: stuffs an MD5 hash into 128 bits
Type 4: stuffs random data into 128 bits
Type 5: stuffs an SHA1 hash into 128 bits
Type 6: unofficial idea for sequential UUIDs
Even though most posts are warning people against the use of UUIDs, they are still very popular. This popularity comes from the fact that these values can easily be generated by remote devices, with a very low probability of collision.
This basic example compiles a simple Go program. The naive way on the left results in a 961 MB image. When using a multi-stage build, we copy just the compiled binary which results in a 7 MB image.
# Image size: 7 MB
FROM golang:1.17.5 as builder
WORKDIR /workspace
COPY . .
ENV CGO_ENABLED=0
RUN go get && go build -o main .
FROM scratch
WORKDIR /workspace
COPY --from=builder \
/workspace/main \
/workspace/main
CMD ["/workspace/main"]
Docker introduced multi-stage builds starting from Docker Engine v17.05. This allows us to perform all preparations steps as before, but then copy only the essential files or output from these steps.
Multi-stage builds are great for Dockerfile steps that aren't used at runtime