Observability Solutions Architect
We are seeking a highly skilled Observability Solutions Architect to join our team. As an Observability Solutions Architect, you will be responsible for designing, implementing, and maintaining a robust observability infrastructure using Elastic, Prometheus, and Grafana. You will work closely with our DevOps, IT, and development teams to ensure comprehensive monitoring, alerting, and visualization of our systems.
Key Responsibilities:
* Assess current monitoring and observability setup and identify gaps.
* Design, implement, and upgrade Prometheus-based monitoring solutions in on-premises setups with multi-tenant and support teams design.
* Configure and maintain Grafana dashboards for real-time visualization with multi-tenant and support teams design.
* Integrate Prometheus with other systems and tools (e.g., Loki, Mimir, Tempo, Thanos).
* Design, implement, and upgrade Elastic (ELK Stack) for on-premises setups.
* Develop and document monitoring and logging strategies and best practices.
* Set up alerts and notification mechanisms to preemptively address system issues.
* Train internal staff on the use and maintenance of Prometheus, Grafana, and Elastic.
* Provide ongoing support and improvements to the observability framework.
* Ensure high availability and performance of the monitoring and logging systems.
* Provide stand-by services on a rotation basis during weekends, holidays, and outside of normal working hours.
Required Qualifications & Experience:
* At least 5 years in a similar role.
* Proven experience in deploying and managing Elastic, Prometheus, and Grafana in on-premises setups with multi-tenant and multi-support teams design.
* Strong understanding of observability concepts and best practices, including APM.
* Experience with related technologies (e.g., Kubernetes, Docker, Kibana, Mimir, Loki, Tempo, Thanos, on-premises infrastructure).
* Proficiency in scripting and automation (e.g., Bash, Python).
* DevOps experience and practice.
* Familiarity with infrastructure-as-code tools (e.g., Ansible, Terraform).
* Experience with log management and tracing solutions (e.g., Loki, ELK stack, Jaeger).
* Knowledge of other monitoring tools is desirable, especially SCOM and Checkmk.
* Programming skills are desirable, especially .NET C# and Python.
Education and Certifications:
* Bachelor's or master's degree in information technology is desirable.
* Monitoring certifications in SCOM, Checkmk, Elastic, Prometheus, Grafana is desirable.