Brief summary

Authress avoided downtime during the AWS us-east-1 outage by implementing a multi-region, redundant infrastructure with automated DNS failover using custom health checks, edge-optimized routing, and robust anomaly detection, backed by rigorous testing and incremental deployments to minimize risk and impact. Their system design assumes failure is inevitable and focuses on quick detection, seamless failover, and minimizing single points of failure through automation and continuous validation.

Long summary

• Authress experienced a major AWS us-east-1 outage affecting DynamoDB and other critical AWS services.
• They run infrastructure in us-east-1 due to customer location demands, despite known risks.
• AWS services like CloudFront, Certificate Manager, Lambda@Edge, and IAM control planes are centralized in us-east-1, impacting availability during incidents.
• Aiming for a 5-nines SLA (99.999% uptime) requires more than relying on AWS SLAs alone, which are insufficient.
• Simple single-region architectures fail to meet high reliability due to frequent AWS incidents.
• Authress recognizes "everything fails all the time" and designs systems assuming failure.
• Retry strategies are mathematically analyzed; third-party components must have at least 99.7% reliability to be usable.
• Multi-region redundant infrastructure with DNS failover via AWS Route 53 health checks enables automatic failover.
• Custom health checks validate actual service health beyond default DNS checks.
• Edge-optimized architecture using CloudFront and Lambda@Edge improves latency and provides better failover options.
• DynamoDB Global Tables replicate data across regions to support failover.
• Rigorous testing and validation, including application-level tests, mitigate risks of bugs in production.
• Incremental deployment (customer buckets) limits impact by rolling out changes gradually.
• Asynchronous validation tests check consistency across databases after deployments.
• Anomaly detection is used to identify meaningful incidents impacting business logic, beyond mere HTTP error codes.
• Customer support feedback is integrated into incident detection to catch undetected or gray failures.
• Security measures include rate limiting, AWS WAF with IP reputation lists, and blocking suspicious high-volume requests.
• Resource exhaustion prevention is critical, with rate limiting implemented at multiple infrastructure layers.
• Infrastructure as Code (IaC) deployment differences across regions and edge leads to challenges in consistency.
• Despite all these measures, achieving a true 5-nines SLA is extremely challenging but remains a core commitment.

Summary of HN discussion

https://news.ycombinator.com/item?id=45955565

• The discussion highlights concerns about automation and Infrastructure as Code (IaC) being potential failure points, emphasizing the challenge of safely updating these systems.
• Rollbacks are rarely automatic; often, knowing in advance to avoid certain rollouts is preferable as automated rollbacks can worsen failures.
• Simple, less complex infrastructure changes are preferred to reduce human error, which is the leading cause of incidents.
• There is skepticism about the reliability of Route 53 failover in practice, with concerns about its failure modes and the complexity of multi-cloud DNS failover.
• Some contributors suggest modular IaC approaches (Pulumi, Terragrunt) for safer, repeatable deployments but warn about added complexity.
• Retry logic in failures is criticized; retries may not improve reliability linearly due to correlated failures and overall system overload during outages.
• Latency and client timeout constraints limit the practical number of retries possible.
• DNS is acknowledged as a single point of failure with caching and failover timing challenges.
• Multi-cloud failover at DNS level is complex, costly, and not widely implemented due to infrastructure and coordination requirements.
• Gray failures (where the system reports healthy but customers experience issues) and the difficulty in knowing real incident impact without customer feedback are noted.
• Customer support is critical in incident detection since automated systems cannot catch every failure.
• Detailed monitoring via CloudFront and telemetry helps identify actual service issues during outages.
• Overall, the theme is the difficulty in achieving perfect reliability, the importance of simplicity, and the need for layered detection and response strategies to manage failures.

• Users appreciate Grafana's rich features but criticize its complexity and brittleness.
• Tools like Prometheus and Grafana are seen as over-engineered, especially with components like Mimir requiring dedicated infrastructure.
• Concerns exist about the future of self-hosted Grafana due to frequent breaking changes.
• Some suggest simpler, stable alternatives for monitoring such as Prometheus alone, Zabbix, VictoriaMetrics, or newer tools like SigNoz, OpenObserve, and Perses.
• The community desires a stable, reliable "boring" monitoring stack rather than constant feature churn.

Overall, there is notable dissatisfaction with Grafana’s direction and operational demands, particularly among smaller teams or startups seeking sustainable solutions.

• Henrik Gerdes warns that Grafana, despite being powerful, has become maintenance-heavy and complex, especially due to recent additions like Mimir and Kubernetes dependence.
• Stability issues, frequent breaking changes, and high operational overhead make it challenging for startups and teams needing reliable monitoring.
• Founders should consider total ownership costs, including updates, migrations, and support, before adopting Grafana.
• Grafana’s flexibility and scalability come with complexity that can erode trust in production environments.

• AWS EKS costs can escalate due to massive, parallel workloads in life sciences/drug development (e.g., genomic sequencing, molecular modeling).
• Default Kubernetes scheduler uses leastAllocated strategy, spreading pods across many nodes for fairness/high availability.
• leastAllocated strategy causes many partially utilized nodes, preventing autoscalers from scaling down idle nodes, increasing costs.
• mostAllocated scheduling strategy "packs" pods onto fewer nodes, maximizing utilization and enabling autoscalers like Karpenter to remove idle nodes.
• Switching to mostAllocated can reduce runtime costs significantly (e.g., ~10% in UAT, 43% in PROD environments).
• Custom scheduler deployment on AWS EKS requires creating a service account, ClusterRoleBindings, RoleBinding, a ConfigMap with the mostAllocated scoring strategy, and a deployment with a matching Kubernetes version container image.
• Resource weights can prioritize packing of expensive resources (e.g., high weight on GPUs for ML workloads).
• Testing in non-production environments is recommended before full rollout.
• Implementing mostAllocated scheduling can dramatically optimize costs by enabling cluster autoscalers to shut down unused nodes.

Summary: Navigating Failures in Pods With Devices

This article examines the unique challenges Kubernetes faces in managing specialized hardware (e.g., GPUs, accelerators) within AI/ML workloads, and explores current pain points, DIY solutions, and the future roadmap for more robust device failure handling.

Why AI/ML Workloads Are Different

• Heavy Dependence on Specialized Hardware: AI/ML jobs require devices like GPUs, with hardware failures causing significant disruptions.
• Complex Scheduling: Tasks may consume entire machines or need coordinated scheduling across nodes due to device interconnects.
• High Running Costs: Specialized nodes are expensive; idle time is wasteful.
• Non-Traditional Failure Models: Standard Kubernetes assumptions (like treating nodes as fungible, or pods as easily replaceable) don’t apply well; failures can trigger large-scale restarts or job aborts.

Major Failure Modes in Kubernetes With Devices

1. Kubernetes Infrastructure Failures

   • Multiple actors (device plugin, kubelet, scheduler) must work together; failures can occur at any stage.
   • Issues include pods failing admission, poor scheduling, or pods unable to run despite healthy hardware.
   • Best Practices: Early restarts, close monitoring, canary deployments, use of verified device plugins and drivers.

2. Device Failures

   • Kubernetes has limited built-in ability to handle device failures—unhealthy devices simply reduce the allocatable count.
   • Lacks correlation between device failure and pod/container failure.
   • DIY Solutions:
     • Node Health Controllers: Restart nodes if device capacity drops, but these can be slow and blunt.
     • Pod Failure Policies: Pods exit with special codes for device errors, but support is limited and mostly for batch jobs.
     • Custom Pod Watchers: Scripts or controllers watch pod/device status, forcibly delete pods attached to failed devices, prompting rescheduling.

3. Container Code Failures

   • Kubernetes can only restart containers or reschedule pods, with limited expressiveness about what counts as failure.
   • For large AI/ML jobs: Orchestration wrappers restart failed main executables, aiming to avoid expensive full job restart cycles.

4. Device Degradation

   • Not all device issues result in outright failure; degraded performance now occurs more frequently (e.g., one slow GPU dragging down training).
   • Detection and remediation are largely DIY; Kubernetes does not yet natively express "degraded" status.

Current Workarounds & Limitations

• Most device-failure strategies are manual or require high privileges.
• Workarounds are often fragile, costly, or disruptive.
• Kubernetes lacks standardized abstractions for device health and device importance at pod or cluster level.

Roadmap: What’s Next for Kubernetes

SIG Node and Kubernetes community are focusing on:

• Improving core reliability: Ensuring kubelet, device manager, and plugins handle failures gracefully.
• Making Failure Signals Visible: Initiatives like KEP 4680 aim to expose device health at pod status level.
• Integration With Pod Failure Policies: Plans to recognize device failures as first-class events for triggering recovery.
• Pod Descheduling: Enabling pods to be rescheduled off failed/unhealthy devices, even with restartPolicy: Always.
• Better Handling for Large-Scale AI/ML Workloads: More granular recovery, fast in-place restarts, state snapshotting.
• Device Degradation Signals: Early discussions on tracking performance degradation, but no mature standard yet.

Key Takeaway

Kubernetes remains the platform of choice for AI/ML, but device- and hardware-aware failure handling is still evolving. Most robust solutions are still "DIY," but community and upstream investment is underway to standardize and automate recovery and resilience for workloads depending on specialized hardware.

See the summary below this highlight

• Introduction

  • Containers are widely used for scalable applications but face challenges with startup times for large images (e.g., AI/ML workloads).
  • Pulling large images from Amazon Elastic Container Registry (ECR) can take several minutes, impacting performance.
  • Bottlerocket, an AWS open-source Linux OS optimized for containers, offers a solution to reduce container startup time.

• Solution Overview

  • Bottlerocket's data volume feature allows prefetching container images locally, eliminating the need for downloading during startup.
  • Prefetching is achieved by creating an Amazon Elastic Block Store (EBS) snapshot of Bottlerocket's data volume and mapping it to new Amazon EKS nodes.
  • Steps to implement:
  • Spin up an Amazon EC2 instance with Bottlerocket AMI.
  • Pull application images from the repository.
  • Create an EBS snapshot of the data volume.
  • Map the snapshot to Amazon EKS node groups.

• Benefits of Bottlerocket

  • It separates OS and container data volumes, ensuring consistency and security during updates.
  • Prefetched images significantly reduce startup times for large containers.

• Implementation Walkthrough

  • Step 1: Build EBS Snapshot
    • Automate snapshot creation using a script.
    • Prefetch images like Jupyter-PyTorch and Kubernetes pause containers.
    • Export the snapshot ID for use in node group configuration.
  • Step 2: Setup Amazon EKS Cluster
    • Create two node groups:
    • no-prefetch-mng: Without prefetched images.
    • prefetch-mng: With prefetched images mapped via EBS snapshot.
  • Step 3: Deploy Pods
    • Test deployment on both node groups.
    • Prefetched nodes start pods in just 3 seconds, compared to 49 seconds without prefetching.

• Results

  • Prefetching reduced container startup time from 49 seconds to 3 seconds, improving efficiency and user experience.

• Further Enhancements

  • Use Karpenter for automated scaling with Bottlerocket nodes.
  • Automate snapshot creation in CI pipelines using GitHub Actions.

• Cleaning Up

  • Delete AWS resources (EKS cluster, Cloud9 environment, EBS snapshots) to avoid charges after testing.

• Conclusion

  • Bottlerocket's data volume prefetching dramatically enhances container startup performance for large workloads on Amazon EKS.

• Why use Kubernetes

  • Best for running multiple processes/servers/jobs with redundancy and load balancing
  • Enables infrastructure-as-code configuration for service relationships
  • Outsourced infrastructure management via cloud providers (e.g., Google Kubernetes Engine)

• What they use

  • Core resources: Deployments (with rolling updates), Services (ClusterIP/LoadBalancer), CronJobs
  • Configuration: ConfigMaps and Secrets via Pulumi (TypeScript) instead of raw YAML
  • Cautious adoptions: StatefulSets for limited persistence, RBAC only when necessary

• What they avoid

  • Hand-written YAML and Helm charts ("fragility for no gain")
  • Operators, custom resources, service meshes, and most third-party controllers
  • Local k8s stack replication (prefer Docker Compose for local dev)

• Key insights

  • "A human should never wait for a pod" - unsuitable for interactive workloads requiring fast startup
  • Use managed databases/storage for critical data instead of k8s volumes
  • Alternatives like Railway/Render may be better for simpler SaaS apps
  • Recently adopted Ingress controllers for Cloud Armor integration despite initial reservations

• Figma's Initial Infrastructure Challenges:

  • Figma's monolithic architecture struggled with resource allocation inefficiencies and limited scalability.
  • High traffic spikes from collaborative design workflows required more robust solutions for resource autoscaling and failover.

• Why Kubernetes Was Chosen:

  • Kubernetes' container orchestration capabilities promised better resource management and service isolation.
  • Features like Horizontal Pod Autoscaling (HPA), robust networking via Kubernetes Services, and support for StatefulSets made it an ideal fit for Figma’s needs.
  • The platform also wanted better alignment with cloud-native practices and modern CI/CD workflows.

• Incremental Migration Approach:

  • Step 1: Non-Critical Services: Figma migrated stateless services first, allowing experimentation without risking core functionality.
  • Step 2: Custom Tooling: Internal tooling was built to manage Kubernetes manifests and automate Helm chart creation for standardization.
  • Step 3: Stateful Services: For databases and other stateful components, Figma relied on Kubernetes' StatefulSets and persistent volumes (PVs) to ensure data integrity during the migration.
  • Step 4: Observability Enhancements: Kubernetes-native tools like Prometheus and Grafana were integrated to provide detailed metrics and system insights.

• Key Technical Adjustments During Migration:

  • Service Discovery: Transitioned to Kubernetes-native DNS for internal service communication, replacing legacy methods.
  • Load Balancing: Leveraged Kubernetes Ingress and external load balancers (e.g., NGINX or cloud-native solutions) for traffic routing.
  • Networking Complexity: Resolved challenges around multi-cluster networking using Kubernetes CNI plugins like Calico.
  • Resource Management: Used Resource Quotas and Limits to prevent pod overcommitment and optimize cluster utilization.

• Challenges Faced:

  • Stateful Services: Ensuring zero-downtime migration for databases required careful orchestration of PersistentVolumeClaims (PVCs) and StatefulSets.
  • Networking: Handling cross-region traffic and external dependencies required tweaking Kubernetes Ingress configurations.
  • Resource Constraints: Balancing costs and performance involved tuning cluster-autoscaler configurations and evaluating node pool setups.

• Benefits Realized Post-Migration:

  • Scalability: Kubernetes' HPA allowed Figma to scale pods dynamically based on traffic patterns, ensuring consistent performance.
  • Deployment Efficiency: CI/CD pipelines integrated seamlessly with Kubernetes, enabling faster and more reliable rollouts using tools like Argo CD.
  • Reliability: Self-healing capabilities, such as pod restarts and node failover, reduced downtime during failures.
  • Observability: Improved system monitoring with Kubernetes' native metrics server and integrations with Prometheus and Grafana.

• Future Enhancements Planned:

  • Service Mesh Integration: Adoption of Istio or Linkerd to enhance observability, security (e.g., mutual TLS), and traffic management.
  • Cost Optimization: Further tuning autoscaling policies and resource limits to minimize waste.
  • Edge Improvements: Deploying Kubernetes clusters closer to end-users for reduced latency, potentially using Kubernetes' Cluster Federation.

• Overview of Tesla's Data Infrastructure Challenges:

  • Modern Tesla vehicles generate an enormous volume of telemetry data related to sensor readings, driver behavior, energy consumption, and more.
  • The primary challenge is ingesting, processing, and analyzing this data at scale while maintaining real-time capabilities.

• Kubernetes for Orchestration:

  • Tesla uses Kubernetes to manage containerized microservices across a distributed cloud environment.
  • Kubernetes ensures dynamic scaling to handle fluctuating workloads, providing high availability for critical services.
  • Each microservice is encapsulated in its own container, improving isolation and deployability.

• Kafka for Real-Time Event Streaming:

  • Apache Kafka is the backbone of Tesla’s data pipeline, managing trillions of events daily from globally distributed vehicles.
  • Kafka topics are structured to partition and replicate data efficiently, ensuring fault tolerance and high throughput.
  • Producers (vehicles) send data to Kafka brokers, while consumers (analytics systems, data lakes) process these streams in real-time.

• Data Processing Pipelines:

  • Data from Kafka is ingested into processing systems for real-time analytics, anomaly detection, and predictive maintenance.
  • Stream processing frameworks (e.g., Apache Flink or Kafka Streams) analyze data for immediate feedback.
  • Batch systems handle aggregation and storage in Tesla’s data lake for long-term insights.

• Key Technical Advantages:

  • Scalability: Kubernetes dynamically allocates resources based on the volume of incoming data and computational requirements.
  • Resilience: Kafka’s replication factor ensures that no single broker failure impacts the system.
  • Low Latency: Data streams from Kafka enable Tesla to act on insights in milliseconds, critical for safety and performance monitoring.

• Simplified Management:

  • The platform supports multi-cluster Kubernetes configurations for geographic data segregation.
  • A central control plane monitors system health, manages deployments, and ensures compliance with data regulations.

• Future Goals and Improvements:

  • Enhancing AI-driven analytics to derive deeper insights from vehicle data.
  • Further optimizing Kafka’s cluster topology to improve fault tolerance and reduce operational costs.
  • Expanding edge processing capabilities in vehicles to pre-filter data, reducing bandwidth requirements to the cloud.

• Kubernetes Configuration Drift Issue:

  • Reddit experienced a significant outage on March 13, 2022, during a Kubernetes upgrade from version 1.23 to 1.24.
  • The outage was caused by configuration drift, where unintended changes accumulated over time, leading to inconsistencies across clusters.

• New Platform Abstraction with Declarative APIs:

  • Reddit adopted a declarative approach, leveraging Kubernetes controllers to manage configurations and enforce consistency.
  • The implementation of these controllers enabled Reddit to abstract platform complexities and ensure a uniform deployment environment.

• Centralized Control Plane for Multi-Cluster Management:

  • The team built a centralized control plane to manage multiple Kubernetes clusters effectively.
  • Cluster provisioning time was drastically reduced from over 30 hours to approximately 2 hours.
  • Centralization facilitated standardized configurations and reduced operational overhead.

• Development of Achilles SDK:

  • Achilles, an SDK developed in-house by Reddit, simplified the creation of Kubernetes controllers.
  • It allowed infrastructure engineers to automate and manage resources programmatically without deep Kubernetes expertise.
  • The SDK supported a more proactive approach to problem-solving, preventing drift by design.

• Benefits and Lessons Learned:

  • The new system ensured robust monitoring, minimized manual intervention, and improved scalability.
  • Configuration drift was effectively mitigated, providing a more stable and predictable infrastructure.
  • The experience highlighted the importance of using Kubernetes-native solutions and declarative configurations for managing large-scale deployments.

• Future Goals:

  • Further refinement of the platform to address edge cases and improve developer experience.
  • Continued investment in tools and processes to maintain infrastructure consistency at scale.

MLOps prediction for 2025

MLOps prediction for 2025

MLOps prediction for 2025

RAG will shine even more in 2025

The introduction of the Image Volume Source feature in Kubernetes 1.31 allows MLOps practitioners to mount OCI-compatible artifacts, such as large language model weights or machine learning models, directly into pods without embedding them in container images. This streamlines model deployment, enhances efficiency, and leverages OCI distribution mechanisms for effective model management.

In essence, this article is about:

1) Training a sample model and uploading it to an S3 bucket:

```python from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression import joblib

Load the Iris dataset

iris = load_iris() X, y = iris.data, iris.target

Split the data into training and testing sets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Train the logistic regression model

model = LogisticRegression(max_iter=200) model.fit(X_train, y_train)

Save the trained model to a file

joblib.dump(model, 'model.pkl') ```

1. Creating a sample Zappa config, because AWS Lambda doesn’t natively support Flask, we need to use Zappa, a tool that helps deploy WSGI applications (like Flask) to AWS Lambda:

```json { "dev": { "app_function": "app.app", "exclude": [ "boto3", "dateutil", "botocore", "s3transfer", "concurrent" ], "profile_name": null, "project_name": "flask-test-app", "runtime": "python3.10", "s3_bucket": "zappa-31096o41b" },

"production": {
    "app_function": "app.app",
    "exclude": [
        "boto3",
        "dateutil",
        "botocore",
        "s3transfer",
        "concurrent"
    ],
    "profile_name": null,
    "project_name": "flask-test-app",
    "runtime": "python3.10",
    "s3_bucket": "zappa-31096o41b"
}

} ```

1. Writing a sample Flask app:

```python import boto3 import joblib import os

Initialize the Flask app

app = Flask(name)

S3 client to download the model

s3 = boto3.client('s3')

Download the model from S3 when the app starts

s3.download_file('your-s3-bucket-name', 'model.pkl', '/tmp/model.pkl') model = joblib.load('/tmp/model.pkl')

@app.route('/predict', methods=['POST']) def predict(): # Get the data from the POST request data = request.get_json(force=True)

# Convert the data into a numpy array
input_data = np.array(data['input']).reshape(1, -1)

# Make a prediction using the model
prediction = model.predict(input_data)

# Return the prediction as a JSON response
return jsonify({'prediction': int(prediction[0])})

if name == 'main': app.run(debug=True) ```

1. Deploying this app to production (to AWS):

bash zappa deploy production

and later eventually updating it:

bash zappa update production

1. We should get a URL like this:

https://xyz123.execute-api.us-east-1.amazonaws.com/production

which we can query:

curl -X POST -H "Content-Type: application/json" -d '{"input": [5.1, 3.5, 1.4, 0.2]}' https://xyz123.execute-api.us-east-1.amazonaws.com/production/predict

The blog post by Cliff Malmborg from Loft Labs discusses optimizing Kubernetes costs using multi-tenancy and virtual clusters. With Kubernetes expenses rising rapidly at scale, traditional cost-saving methods like autoscaling, resource quotas, and monitoring tools help but are not enough for complex environments where underutilized clusters are common. Multi-tenancy enables resource sharing, reducing the number of clusters and, in turn, management and operational costs.

A virtual cluster is a fully functional Kubernetes cluster running within a larger host cluster, providing better isolation and flexibility than namespaces. Unlike namespaces, each virtual cluster has its own Kubernetes control plane, so resources like statefulsets and webhooks are isolated within it, while only core resources (like pods and services) are shared with the host cluster. This setup addresses the "noisy neighbor" problem, where workloads in a shared environment interfere with each other due to resource contention.

Virtual clusters offer the isolation benefits of individual physical clusters but are cheaper and easier to manage than deploying separate physical clusters for each tenant or application. They also support "sleep mode," automatically scaling down unused resources to save costs, and allow shared use of central tools (like ingress controllers) installed in the host cluster. By transitioning to virtual clusters, companies can balance security, isolation, and cost-effectiveness, reducing the need for multiple physical clusters and making Kubernetes infrastructure scalable for modern, resource-demanding applications.

They went through 310 role descriptions and, even though role descriptions may vary significantly, they found 3 core skills that a large percentage of MLOps roles required:

📦 Docker and Kubernetes 🐍 Python 🌥 Cloud

Kubeflow sounds like the most feature rich solution, whose main con is its UI and the setup process

Summary of pros/cons of Airflow, Kubeflow and Prefect

Main con of Airflow

Prefect

(see the graph above)

Responsibility of the orchestration tool

Business ¯\_(ツ)_/¯

ServingRuntime

Kubernetes clusters have a maximum IP address limitation

If I am not mistaken, the multi-model approach reduces the size by 90% in this case

Benefits of multi-model serving

With more separation, comes the problem of distribution

https://mlflow.org/docs/latest/projects.html

Sample metrics to monitor

Example above shows how to upload an image to an API endpoint with FastAPI.

Example below is a bit more complex.

first mention of LLMops I've seen in the wild

Command to check whether an action is allowed

tl;dw (best DevOps tools in 2023)

1. Low-budget cloud computing : Civo (close to Scaleway)
2. Infrastructure and Service Management: Crossplane
3. App Management - manifests : cdk8s (yes, not Kustomize or Helm)
4. App Management - k8s operators: tie between Knative and Crossplane
5. App Management - managed services: Google Cloud Run
6. Dev Envs: Okteto (yeap, not GitPod)
7. CI/CD: GitHub Actions (as it's simplest to use)
8. GitOps (CD): Argo CD (wins with Flux due to its adoption rate)
9. Policy Management: Kyverno (simpler to use than industry's most powerful tool: OPA / Gatekeeper)
10. Observability: OpenTelemetry (instrumentation of apps), VictoriaMetrics (metrics - yes not Prometheus), Grafana / Loki (logs), Grafana Tempo (tracing), Grafana (dashboards), Robusta (alerting), Komodor (troubleshooting)

https://rstudio.github.io/vetiver-r/

Consider the following packages while doing MLOps in R: - pins - vetiver - tidymodels - tidyverse

mlflow.evaluate()

AutoML in MLflow 2.0

MLflow Recipes in MLflow 2.0

List of differences between a microservice and inference service.

(see bullet points below annotation)

My research on choosing a DAG library to extend, ca. Autumn 2019.

Real project demoing MLflow & DVC, Prometheus & Grafana.

Real project combining MLflow & DVC.

Voting for ClearML with a video.

Infos in the comments about DVC MLOps & one suggesting ClearML.

-Stars history comparing MLOps platforms: - Airflow - mlflow - Argo-workflows - DVC - pachyderm - ClearML - Kubeflow - Luigi

Combine the use of DVC & mlflow.

Multi-faced comparisons, but not all of them #FOSS.

GitHub stars evolution for the top-contenders described in this tweet.

Textual description of by-pair comparisons of the top contenders of the ETL/Pipelne/ML space.

Who really is MLOps Engineer ;)

Recommended MLOps team structure

18 MLOps FOSS tools. The Q remains: how some of these combine all together?

Explain the need for [[mlflow]] and more tools for data/model governance.

Somewhat biased for neptune, omitting things unsupported on its side,

Comparing:

Experiment-tracking alone tools:

• neptune.ai
• Wandb

Full-lifecycle tools:

• MLflow/Databricks ...

Analyze pros & cons based on the the well-written distinction between these roles: - Data scientist/Researcher - ML Engineer - Project Lead

Neptune.io PoV.

How "Staging" environment can be useful

Main disadvantage of managed MLOps platforms

4 CI free tier comparison: * Quality of Documentation * Compute Power * Available Disk Space * Free Build Minutes * Speed and Performance

About Docker Engine and Docker Desktop

Containers (the backend):

See sample Dockerfile above this annotation (below there is a version tweaked even further)

Keeping staging environment has its cost

Podman 4.0 can transfer container images without a registry.

For example: * You can copy a root image to a non-root account:

$ podman image scp root@localhost::IMAGE USER@localhost:: * Or copy an image from one machine to another with this command:

$ podman image scp me@192.168.68.122::IMAGE you@192.168.68.128::

Prefer PATCH over PUT

For example: ``` // Request => GET /users/4TL011ax

// Response <= 404 Not Found { "code": "user/not_found", "message": "A user with the ID 4TL011ax could not be found." } ```

2 examples of versioning APIs

https://en.wikipedia.org/wiki/ISO_8601

Problem with Poetry/Docker

Approach of using poetry-dynamic-versioning plugin

Be careful when updating the version field of pyproject.toml around Docker

How VCR.py works

VCR.py library to speed up Python HTTP tests

3 things MLOps can learn from DevOps

State of MLOps in March 2022

Quarkus framework

Kubernetes-native application

Cloud-native applications

Case of Salesforce serving 100K-500K ML models with the use of shuffle sharding

Barching predictions

Responsibilities of Inference Service

Contents of a model config file

MLOps focuses deeply on automation and standardization

Shadow mode

It's possible to mount FastAPI applications within a FastAPI application

There are 5 types of UUIDs (source):

Type 1: stuffs MAC address+datetime into 128 bits

Type 3: stuffs an MD5 hash into 128 bits

Type 4: stuffs random data into 128 bits

Type 5: stuffs an SHA1 hash into 128 bits

Type 6: unofficial idea for sequential UUIDs

Typical random issues when deploying microservices

Microservices have quite many moving parts

9 things needed for deploying a microservice (listed below)

5 things needed for deploying a monolith (listed below)

Pros of microservices (not always all are applicable):

• Fault isolation
• Eliminating the technology lock
• Easier understanding
• Faster deployment
• Scalability

python:3.9-slim-bullseye may be the sweet spot for a Python Docker image

Choosing the best Python base Docker image depends on different factors.

3 candidates for the best Python base Docker image

kubectl run a –image alpine –command — /bin/sleep 1d

kubectl get pods -A -o wide

kubectl get nodes -o wide

kubectl get componentstatus

kubectl get events -A

kubectl api-resources -o wide –sort-by name

Alternative to kubectl get --raw '/healthz?verbose'. It does not show scheduler or controller-manager output, but it adds a lot of additional checks that might be valuable if things are broken

8 commands to debug Kubernetes cluster:

kubectl version --short
kubectl cluster-info
kubectl get componentstatus
kubectl api-resources -o wide --sort-by name
kubectl get events -A
kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl run a --image alpine --command -- /bin/sleep 1d

Current top orchestrators:

• Airflow
• Argo
• Google Cloud Composer
• AWS Step Functions

Requirements specific to MLOps systems:

1. Large scale of operations
2. Orchestration
3. Robust versioning (data, models, code)
4. Apps integrated to surrounding busness systems

One of the best ways to picture a difference between DevOps and MLOps

High level definition of Argo Workflow

Pros of Argo Workflow:

• Resilience
• Autoscaling
• Configurability
• Support for RBAC

Cons of Argo Workflow:

• A lot of YAML files required
• k8s knowledge required

Should you go into Argo, or not?

kind: WorkflowTemplate

(have a look below this highlight for a full reasoning)

Summary of data tools in October 2021: http://46eybw2v1nh52oe80d3bi91u-wpengine.netdna-ssl.com/wp-content/uploads/2021/09/ML-AI-Data-Landscape-2021.pdf

KFServing is now KServe

At the current moment the best approach is to use minikube with a preferred backend (Docker Engine and Podman are already there), and you can simply run one command to configure Docker CLI to use the engine from the cluster.

• Drivers | minikube
• docker-env | minikube

k3d <--- k3s that allows to run mult-node clusters on a local machine

KinD (Kubernetes in Docker) <--- sounds like the most recommended solution to learn k8s locally

k0s <--- similar to k3s, but not as lightweight

k3s <--- lightweight solution

7 tools for learning k8s locally:

1. k3s
2. k0s
3. Microk8s
4. DinD
5. minikube
6. KinD
7. k3d

We can run Kubernetes locally as a:

1. binary package
2. container using dind

With arkade, we can quickly set up different k8s tools, while using a single command:

e.g. arkade get k9s

It turns out that this phrase doesn't lead to an existing research. If one goes down the rabbit hole, it all ends up with dead links

Vital components of a pipeline

FastAPI does not include a web server like Flask. Therefore, it requires Uvicorn.

Not having a web server has pros and cons listed here

Testing server latency with curl:

1) Get this file from GitHub

2) Run the curl: curl -L -w "@curl-format.txt" -o tmp -s $YOUR_URL

To test a failing Dockerfile step, it is best to comment it out, successfully build an image, and then run this command from inside of the Dockerfile

4 different options to install Poetry through a Dockerfile

Log inputs and outputs of your online models to prevent training-serving skew

Encourage idempotency