{"id":4543,"date":"2025-05-19T10:54:43","date_gmt":"2025-05-19T08:54:43","guid":{"rendered":"https:\/\/datalchemy.net\/nos-prestations\/data-architecture-monitoring-and-dashboards\/"},"modified":"2025-08-19T10:18:45","modified_gmt":"2025-08-19T08:18:45","slug":"data-architecture-monitoring-and-dashboards","status":"publish","type":"projet","link":"https:\/\/datalchemy.net\/en\/our-services\/data-architecture-monitoring-and-dashboards\/","title":{"rendered":"Data architecture, monitoring and dashboards"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\"><strong>Context<\/strong><\/h2>\n\n<ul class=\"wp-block-list\">\n<li>Data audits carried out upstream by Datalchemy validated the feasibility of exploiting \u00c9cole 42 data for analysis, reporting and, ultimately, machine learning.<\/li>\n\n\n\n<li>The first step is to set up a data architecture that enables us to extract a clean, controlled subset of production, without impacting online systems, and to build up a usable history.<\/li>\n\n\n\n<li>The priority need identified is the overhaul of campus dashboards (replacing existing Google reports), with an ambitious deadline set for February 2024.<\/li>\n<\/ul>\n\n<h2 class=\"wp-block-heading\"><strong>Issues<\/strong><\/h2>\n\n<ul class=\"wp-block-list\">\n<li>Very large volumes (data collected on all campuses worldwide).<\/li>\n\n\n\n<li>Limited history: backups overwrite previous data.<\/li>\n\n\n\n<li>Lack of validation and control over archived data.<\/li>\n<\/ul>\n\n<h2 class=\"wp-block-heading\"><strong>Project objectives<\/strong><\/h2>\n\n<ul class=\"wp-block-list\">\n<li>Design an automated architecture to extract data from multiple sources.<\/li>\n\n\n\n<li>Propose a simple system for defining and launching new extractions.<\/li>\n\n\n\n<li>Track the evolution of data over time (versioning, time stamping).<\/li>\n\n\n\n<li>Implement control and validation mechanisms for extracted elements.<\/li>\n\n\n\n<li>Facilitate feedback and management of errors detected during the pipeline.<\/li>\n<\/ul>\n\n<h2 class=\"wp-block-heading\"><strong>Completed work<\/strong><\/h2>\n\n<ul class=\"wp-block-list\">\n<li>Setting up dedicated S3 buckets :\n<ul class=\"wp-block-list\">\n<li>for dumps with automatic file rotation,<\/li>\n\n\n\n<li>for workflow data.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Creation of DVC repositories in production (with management of alternative branches) for dataset versioning.<\/li>\n\n\n\n<li>Deployment of a PostgreSQL database dedicated to monitoring extraction workflows.<\/li>\n\n\n\n<li>Provisioning a Kubernetes cluster with  \n<ul class=\"wp-block-list\">\n<li>Isolated namespace<\/li>\n\n\n\n<li>Pipeline orchestration via Argo Workflows and Cron.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n<h2 class=\"wp-block-heading\"><strong>Results<\/strong><\/h2>\n\n<ul class=\"wp-block-list\">\n<li>Operational infrastructure ready to feed new campus dashboards.<\/li>\n\n\n\n<li>Reproducible and extensible pipeline for other use cases (reporting, ML).<\/li>\n\n\n\n<li>Guaranteed data traceability and control thanks to DVC versioning and workflow logs.<\/li>\n\n\n\n<li>A solid foundation for later deployment of advanced analysis and predictive models.<\/li>\n<\/ul>\n","protected":false},"featured_media":3952,"template":"","meta":{"_acf_changed":false,"inline_featured_image":false},"tags":[205,228,185,204,187,183,184],"class_list":["post-4543","projet","type-projet","status-publish","has-post-thumbnail","hentry","tag-argo-workflow-en","tag-dvc-en","tag-fastapi-en","tag-kubernetes-en","tag-postgresql-en","tag-python-en","tag-s3-en"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v25.8 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Data architecture, monitoring and dashboards - Datalchemy<\/title>\n<meta name=\"description\" content=\"For 42, we designed and deployed a robust and scalable data architecture.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/datalchemy.net\/en\/our-services\/data-architecture-monitoring-and-dashboards\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Data architecture, monitoring and dashboards - Datalchemy\" \/>\n<meta property=\"og:description\" content=\"For 42, we designed and deployed a robust and scalable data architecture.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/datalchemy.net\/en\/our-services\/data-architecture-monitoring-and-dashboards\/\" \/>\n<meta property=\"og:site_name\" content=\"Datalchemy\" \/>\n<meta property=\"article:modified_time\" content=\"2025-08-19T08:18:45+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/datalchemy.net\/wp-content\/uploads\/2025\/05\/archi-data-pour-42.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1024\" \/>\n\t<meta property=\"og:image:height\" content=\"1024\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/datalchemy.net\/en\/our-services\/data-architecture-monitoring-and-dashboards\/\",\"url\":\"https:\/\/datalchemy.net\/en\/our-services\/data-architecture-monitoring-and-dashboards\/\",\"name\":\"Data architecture, monitoring and dashboards - Datalchemy\",\"isPartOf\":{\"@id\":\"https:\/\/datalchemy.net\/en\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/datalchemy.net\/en\/our-services\/data-architecture-monitoring-and-dashboards\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/datalchemy.net\/en\/our-services\/data-architecture-monitoring-and-dashboards\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/datalchemy.net\/wp-content\/uploads\/2025\/05\/archi-data-pour-42.png\",\"datePublished\":\"2025-05-19T08:54:43+00:00\",\"dateModified\":\"2025-08-19T08:18:45+00:00\",\"description\":\"For 42, we designed and deployed a robust and scalable data architecture.\",\"breadcrumb\":{\"@id\":\"https:\/\/datalchemy.net\/en\/our-services\/data-architecture-monitoring-and-dashboards\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/datalchemy.net\/en\/our-services\/data-architecture-monitoring-and-dashboards\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/datalchemy.net\/en\/our-services\/data-architecture-monitoring-and-dashboards\/#primaryimage\",\"url\":\"https:\/\/datalchemy.net\/wp-content\/uploads\/2025\/05\/archi-data-pour-42.png\",\"contentUrl\":\"https:\/\/datalchemy.net\/wp-content\/uploads\/2025\/05\/archi-data-pour-42.png\",\"width\":1024,\"height\":1024,\"caption\":\"Architecture Data pour 42\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/datalchemy.net\/en\/our-services\/data-architecture-monitoring-and-dashboards\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/datalchemy.net\/en\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Projects\",\"item\":\"https:\/\/datalchemy.net\/en\/our-services\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Data architecture, monitoring and dashboards\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/datalchemy.net\/en\/#website\",\"url\":\"https:\/\/datalchemy.net\/en\/\",\"name\":\"Datalchemy\",\"description\":\"Expertise, accompagnement  et R&amp;D en data et IA\",\"publisher\":{\"@id\":\"https:\/\/datalchemy.net\/en\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/datalchemy.net\/en\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/datalchemy.net\/en\/#organization\",\"name\":\"Datalchemy\",\"url\":\"https:\/\/datalchemy.net\/en\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/datalchemy.net\/en\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/datalchemy.net\/wp-content\/uploads\/2025\/08\/logo-datalchemy.gif\",\"contentUrl\":\"https:\/\/datalchemy.net\/wp-content\/uploads\/2025\/08\/logo-datalchemy.gif\",\"width\":696,\"height\":696,\"caption\":\"Datalchemy\"},\"image\":{\"@id\":\"https:\/\/datalchemy.net\/en\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.linkedin.com\/company\/sas-datalchemy\/\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Data architecture, monitoring and dashboards - Datalchemy","description":"For 42, we designed and deployed a robust and scalable data architecture.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/datalchemy.net\/en\/our-services\/data-architecture-monitoring-and-dashboards\/","og_locale":"en_US","og_type":"article","og_title":"Data architecture, monitoring and dashboards - Datalchemy","og_description":"For 42, we designed and deployed a robust and scalable data architecture.","og_url":"https:\/\/datalchemy.net\/en\/our-services\/data-architecture-monitoring-and-dashboards\/","og_site_name":"Datalchemy","article_modified_time":"2025-08-19T08:18:45+00:00","og_image":[{"width":1024,"height":1024,"url":"https:\/\/datalchemy.net\/wp-content\/uploads\/2025\/05\/archi-data-pour-42.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/datalchemy.net\/en\/our-services\/data-architecture-monitoring-and-dashboards\/","url":"https:\/\/datalchemy.net\/en\/our-services\/data-architecture-monitoring-and-dashboards\/","name":"Data architecture, monitoring and dashboards - Datalchemy","isPartOf":{"@id":"https:\/\/datalchemy.net\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/datalchemy.net\/en\/our-services\/data-architecture-monitoring-and-dashboards\/#primaryimage"},"image":{"@id":"https:\/\/datalchemy.net\/en\/our-services\/data-architecture-monitoring-and-dashboards\/#primaryimage"},"thumbnailUrl":"https:\/\/datalchemy.net\/wp-content\/uploads\/2025\/05\/archi-data-pour-42.png","datePublished":"2025-05-19T08:54:43+00:00","dateModified":"2025-08-19T08:18:45+00:00","description":"For 42, we designed and deployed a robust and scalable data architecture.","breadcrumb":{"@id":"https:\/\/datalchemy.net\/en\/our-services\/data-architecture-monitoring-and-dashboards\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/datalchemy.net\/en\/our-services\/data-architecture-monitoring-and-dashboards\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/datalchemy.net\/en\/our-services\/data-architecture-monitoring-and-dashboards\/#primaryimage","url":"https:\/\/datalchemy.net\/wp-content\/uploads\/2025\/05\/archi-data-pour-42.png","contentUrl":"https:\/\/datalchemy.net\/wp-content\/uploads\/2025\/05\/archi-data-pour-42.png","width":1024,"height":1024,"caption":"Architecture Data pour 42"},{"@type":"BreadcrumbList","@id":"https:\/\/datalchemy.net\/en\/our-services\/data-architecture-monitoring-and-dashboards\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/datalchemy.net\/en\/"},{"@type":"ListItem","position":2,"name":"Projects","item":"https:\/\/datalchemy.net\/en\/our-services\/"},{"@type":"ListItem","position":3,"name":"Data architecture, monitoring and dashboards"}]},{"@type":"WebSite","@id":"https:\/\/datalchemy.net\/en\/#website","url":"https:\/\/datalchemy.net\/en\/","name":"Datalchemy","description":"Expertise, accompagnement  et R&amp;D en data et IA","publisher":{"@id":"https:\/\/datalchemy.net\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/datalchemy.net\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/datalchemy.net\/en\/#organization","name":"Datalchemy","url":"https:\/\/datalchemy.net\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/datalchemy.net\/en\/#\/schema\/logo\/image\/","url":"https:\/\/datalchemy.net\/wp-content\/uploads\/2025\/08\/logo-datalchemy.gif","contentUrl":"https:\/\/datalchemy.net\/wp-content\/uploads\/2025\/08\/logo-datalchemy.gif","width":696,"height":696,"caption":"Datalchemy"},"image":{"@id":"https:\/\/datalchemy.net\/en\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.linkedin.com\/company\/sas-datalchemy\/"]}]}},"_links":{"self":[{"href":"https:\/\/datalchemy.net\/en\/wp-json\/wp\/v2\/projet\/4543","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/datalchemy.net\/en\/wp-json\/wp\/v2\/projet"}],"about":[{"href":"https:\/\/datalchemy.net\/en\/wp-json\/wp\/v2\/types\/projet"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/datalchemy.net\/en\/wp-json\/wp\/v2\/media\/3952"}],"wp:attachment":[{"href":"https:\/\/datalchemy.net\/en\/wp-json\/wp\/v2\/media?parent=4543"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/datalchemy.net\/en\/wp-json\/wp\/v2\/tags?post=4543"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}