vmois.dev - personal tech blog https://vmois.dev/ In this part of the MiWaitWay series, we will improve GTFS-rt (real-time) vehicle location ingestor reliability using Write-Ahead-File (WAF) and Sentry monitoring. In the previous part, while transforming raw data, I realized that vehicle location data for some hours was missing. Because of that, it is not possible to correctly... Tue, 04 Jun 2024 00:00:00 +0200 https://vmois.dev/reliable-gtfs-real-time-ingestion-miwaitway/ https://vmois.dev/reliable-gtfs-real-time-ingestion-miwaitway/ In this part of the MiWaitWay series, I transform raw data loaded in the previous article so that it can be used with BigQuery’s powerful GEOGRAPHY functions. To learn more about the MiWaitWay project, check the introduction article. Check the previous article, where I talk about loading raw data into... Mon, 20 May 2024 00:00:00 +0200 https://vmois.dev/transform-raw-data-to-use-bigquery-geography-functions-miwaitway/ https://vmois.dev/transform-raw-data-to-use-bigquery-geography-functions-miwaitway/ In this part of the MiWaitWay series, I will describe how I loaded data from the MiWay website into BigQuery. To learn more about the MiWaitWay project, check the introduction article. What is GTFS data? In the previous articles, I mentioned GTFS data several times. But what is it? General... Mon, 06 May 2024 00:00:00 +0200 https://vmois.dev/ingest-gtfs-data-to-bigquery-miwaitway/ https://vmois.dev/ingest-gtfs-data-to-bigquery-miwaitway/ In this part of the series, I want to share how I imagine data flowing throughout the BigQuery in the MiWaitWay project. To learn more about the MiWaitWay project and its high-level architecture, check the introduction article. You can also find a previous part of the series where I deploy... Thu, 18 Apr 2024 00:00:00 +0200 https://vmois.dev/data-flow-bigquery-miwaitway/ https://vmois.dev/data-flow-bigquery-miwaitway/ In this part of the MiWaitWay series, I focus on configuring Apache Airflow to consume as few resources as possible and deploy it to a small VM on Google Cloud. In addition, I need to leave some resources for a web app that will host a dashboard/map showing the average... Fri, 12 Apr 2024 00:00:00 +0200 https://vmois.dev/deploy-airflow-google-cloud-miwaitway/ https://vmois.dev/deploy-airflow-google-cloud-miwaitway/ For a long time, I wanted to build a project that worked with end-to-end data, from data ingestion and analysis to the final web app that can be shared with users. I like public transport and, recently, have found a topic worth exploring - the average wait time for a... Thu, 04 Apr 2024 00:00:00 +0200 https://vmois.dev/miwaitway-average-wait-time-on-stop/ https://vmois.dev/miwaitway-average-wait-time-on-stop/ Mojo standard library (stdlib) was open-sourced yesterday. It is exciting that the community can now contribute directly to the codebase. After spending some time with the stdlib repository, I want to share how you can change the standard library and make Mojo use your version when running programs. The steps... Fri, 29 Mar 2024 00:00:00 +0100 https://vmois.dev/mojo-local-stdlib-build/ https://vmois.dev/mojo-local-stdlib-build/ In this short article, I want to share some materials and tips about learning Chisel, a hardware description language (HDL) embedded in Scala. It will be handy for software engineers interested in hardware design. Expect this article to be updated as I progress with my learning journey. It is worth... Thu, 13 Jul 2023 00:00:00 +0200 https://vmois.dev/learning-chisel/ https://vmois.dev/learning-chisel/ In this short article, I want to show you how to query the Signal Desktop messages locally using JavaScript. It can be helpful in case you want to build some application or script that, for example, exports or backups Signal messages. Are you a user of the Signal Desktop app?... Sat, 21 Jan 2023 00:00:00 +0100 https://vmois.dev/query-signal-desktop-messages-sqlite/ https://vmois.dev/query-signal-desktop-messages-sqlite/ Code reviews are standard practice in most software companies. It helps to improve code correctness and quality and catch bugs early on. Because you will spend a lot of your software development time on code reviews, being good at it is essential. The code review involves two parties: the reviewer... Sat, 15 Oct 2022 00:00:00 +0200 https://vmois.dev/code-review-tips/ https://vmois.dev/code-review-tips/ In this article, I want to describe two algorithms you can use to find a median on streaming data. Both of them calculate the median in one pass and are storage efficient. Motivation Median (or 50th percentile) is a well-known metric. The “traditional” way of calculating the median is to... Wed, 03 Aug 2022 00:00:00 +0200 https://vmois.dev/find-median-in-streaming-data/ https://vmois.dev/find-median-in-streaming-data/ In this article, I want to write about why and how I built my web analytics from scratch. I will explain my motivation and walk you through the technical details of the project. Why to roll your own analytics? I want to write articles that are useful and insightful. To... Thu, 05 May 2022 00:00:00 +0200 https://vmois.dev/build-web-analytics-project-from-scratch/ https://vmois.dev/build-web-analytics-project-from-scratch/ GitHub Projects is a recently released free GitHub project management tool that provides a flexible way to organize work. Despite that it is still in beta, I think this is a big step forward. But, with great flexibility comes great uncertainty. That is why, in this article, I want to... Sun, 30 Jan 2022 00:00:00 +0100 https://vmois.dev/how-to-use-github-projects-effectively/ https://vmois.dev/how-to-use-github-projects-effectively/ Around 1.5 months ago, I have started learning iOS development using SwiftUI. I am building a small app to keep track of my finances. As part of my learning, I decided to set up a workflow to run Xcode tests using GitHub Actions. And today, I want to share my... Wed, 22 Dec 2021 00:00:00 +0100 https://vmois.dev/xcode-github-actions/ https://vmois.dev/xcode-github-actions/ This short article covers a template for setting up a Flask based web app with a background thread. Knowledge of Python and basic knowledge of Flask is required. Introduction Let’s use an example to demonstrate how to add a background thread to Flask app. Imagine we have a Flask app... Sat, 14 Aug 2021 00:00:00 +0200 https://vmois.dev/python-flask-background-thread/ https://vmois.dev/python-flask-background-thread/ We know that cache can greatly speed up a load of frequently used data (for example, S3 objects). Python 3 has a built-in implementation of simple unbound and LRU caches in functools module called cache and lru_cache respectively (functools documentation). Both cache and lru_cache expose some useful methods like cache_clear... Sun, 20 Jun 2021 00:00:00 +0200 https://vmois.dev/python-timed-lru-cache-methods/ https://vmois.dev/python-timed-lru-cache-methods/