Всем привет. Я хочу рассказать о том, как Sentry восстанавливался после неудачного обновления.
Что это такое Часовой ? Это система отслеживания ошибок с полным открытым исходным кодом, которая поддерживает широкий спектр серверных, браузерных, настольных и мобильных языков и платформ, включая PHP, Node. js, Python, Ruby, C#, Java, Go, React, Angular, Vue, JavaScript и другие.
Немного о том, как просто и естественно все сломалось.
Мы жили на версии 9.0.0, и пришло время обновиться.
Пощупав веб-интерфейс 10.0.0, решил перейти на него.
Это была самая большая ошибка! Обновление произошло как обычно.
Сначала перейди на 9.1.2 потом на 10.0.0 (иначе никак).
Потом обнаружил, что рабочий не запускается.
Причиной этого является добавление новых приложений и зависимостей.
Попытки собрать приложения и закинуть их в к8с не увенчались успехом.
Было принято решение откатиться.
Но здесь меня разочаровало.
После перехода на версию 10 часть данных перемещается в Clickhouse и удаляется из Postgres. После 3 часов работы и привлечения специалиста DBA данные были восстановлены до рабочего состояния.
Как? Сделали рядом инстанс PG, закинули туда схему из версии 9.0.0 и начали потабличное восстановление.
Конечно, здесь были ошибки и файл.
На этом история не заканчивается.
Пришло время обновиться до версии 9.1.2. Я собрал образ и запустил его:
и мы видим это чудесное сообщение:sentry upgrade
! I'm not trusting myself; either fix this yourself by fiddling
! with the south_migrationhistory table, or pass --delete-ghost-migrations
! to South to have it delete ALL of these records (this may not be good).
Exception in thread Thread-1 (most likely raised during interpreter shutdown)
После долгих поисков, проб и ошибок решение было найдено.
Необходимо восстановить схему данных текущей версии.
Для этого клонируем репозиторий: git clone https://github.com/getsentry/onpremise.git
Перейдем к ветке 9.1.2, так как ветки 9.0.0 больше не существует. cd onpremise && git checkout tags/9.1.2
Далее редактируем изображение в Dockerfile: #cat Dockerfile
ARG SENTRY_IMAGE
FROM ${SENTRY_IMAGE:-sentry:9.0.0}-onbuild
Затем мы редактируем файл компоновки docker-compose.yml
# NOTE: This docker-compose.yml is meant to be just an example of how
# you could accomplish this on your own. It is not intended to work in
# all use-cases and must be adapted to fit your needs. This is merely
# a guideline.
# See docs.getsentry.com/on-premise/server/ for full
# instructions
version: '3.4'
x-defaults: &defaults
restart: unless-stopped
build:
context: .
depends_on: - redis # - postgres - memcached - smtp env_file: .
env
environment:
SENTRY_MEMCACHED_HOST: memcached
SENTRY_REDIS_HOST: redis
SENTRY_POSTGRES_HOST: 'sentry.cl.ats'
SENTRY_DB_USER: 'sentryDB'
SENTRY_DB_NAME: 'sentry_user'
SENTRY_DB_PASSWORD: 'sentry_passwd'
SENTRY_POSTGRES_PORT: 5432
SENTRY_EMAIL_HOST: smtp
volumes:
- sentry-data:/var/lib/sentry/files
services:
smtp:
restart: unless-stopped
image: tianon/exim4
memcached:
restart: unless-stopped
image: memcached:1.5-alpine
redis:
restart: unless-stopped
image: redis:3.2-alpine
#postgres:
# restart: unless-stopped
# image: postgres:9.5
# environment:
# POSTGRES_HOST_AUTH_METHOD: 'trust'
# volumes:
# - sentry-postgres:/var/lib/postgresql/data
web:
<<: *defaults
ports:
- '9000:9000'
cron:
<<: *defaults
command: run cron
worker:
<<: *defaults
command: run worker
volumes:
sentry-data:
external: true
#sentry-postgres:
# external: true
Несколько уточнений.
Поскольку у меня есть отдельный экземпляр базы данных, я пропускаю создание через композицию; том сентри-постгрес тоже не нужен.
После этих манипуляций приступаем к сборке: .
/install.sh
В процессе может появиться ошибка, о которой я упоминал выше.
Не обращаем внимания и после завершения процесса выполняем следующую команду: docker-compose run --rm web sentry django migrate --delete-ghost-migrations
Пример вывода Creating network "onpremise_default" with the default driver
Creating onpremise_memcached_1 .
done Creating onpremise_smtp_1 .
done Creating onpremise_redis_1 .
done
21:30:26 [INFO] sentry.plugins.github: apps-not-configured
Running migrations for sentry:
- Nothing to migrate.
- Loading initial data for sentry.
Installed 0 object(s) from 0 fixture(s)
Running migrations for nodestore:
- Nothing to migrate.
- Loading initial data for nodestore.
Installed 0 object(s) from 0 fixture(s)
Running migrations for search:
- Nothing to migrate.
- Loading initial data for search.
Installed 0 object(s) from 0 fixture(s)
Running migrations for social_auth:
- Nothing to migrate.
- Loading initial data for social_auth.
Installed 0 object(s) from 0 fixture(s)
Running migrations for tagstore:
- Nothing to migrate.
- Loading initial data for tagstore.
Installed 0 object(s) from 0 fixture(s)
Running migrations for jira_ac:
- Nothing to migrate.
- Loading initial data for jira_ac.
Installed 0 object(s) from 0 fixture(s)
Running migrations for hipchat_ac:
- Nothing to migrate.
- Loading initial data for hipchat_ac.
Installed 0 object(s) from 0 fixture(s)
Затем вы можете запустить обновление.
docker-compose run --rm web sentry upgrade
Стандартный вывод Starting onpremise_smtp_1 .
done Starting onpremise_memcached_1 .
done Starting onpremise_redis_1 .
done 21:30:48 [INFO] sentry.plugins.github: apps-not-configured Syncing. Creating tables .
Installing custom SQL .
Installing indexes .
Installed 0 object(s) from 0 fixture(s) Synced: > django.contrib.admin > django.contrib.auth > django.contrib.contenttypes > django.contrib.messages > django.contrib.sessions > django.contrib.sites > django.contrib.staticfiles > crispy_forms > debug_toolbar > raven.contrib.django.raven_compat > rest_framework > sentry.plugins.sentry_interface_types > sentry.plugins.sentry_mail > sentry.plugins.sentry_urls > sentry.plugins.sentry_useragents > sentry.plugins.sentry_webhooks > sudo > south > sentry_plugins.slack Not synced (use migrations): - sentry - sentry.nodestore - sentry.search - social_auth - sentry.tagstore - sentry_plugins.jira_ac - sentry_plugins.hipchat_ac (use .
/manage.py migrate to migrate these)
Running migrations for sentry:
- Nothing to migrate.
- Loading initial data for sentry.
Installed 0 object(s) from 0 fixture(s)
Running migrations for nodestore:
- Nothing to migrate.
- Loading initial data for nodestore.
Installed 0 object(s) from 0 fixture(s)
Running migrations for search:
- Nothing to migrate.
- Loading initial data for search.
Installed 0 object(s) from 0 fixture(s)
Running migrations for social_auth:
- Nothing to migrate.
- Loading initial data for social_auth.
Installed 0 object(s) from 0 fixture(s)
Running migrations for tagstore:
- Nothing to migrate.
- Loading initial data for tagstore.
Installed 0 object(s) from 0 fixture(s)
Running migrations for jira_ac:
- Nothing to migrate.
- Loading initial data for jira_ac.
Installed 0 object(s) from 0 fixture(s)
Running migrations for hipchat_ac:
- Nothing to migrate.
- Loading initial data for hipchat_ac.
Installed 0 object(s) from 0 fixture(s)
Creating missing DSNs
Correcting Group.num_comments counter
Итак, схема восстановлена.
Вы можете обновиться до 9.1.2.
Для этого измените версию в Dockerfile на 9.1.2: ARG SENTRY_IMAGE
FROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuild
И запустите install.sh .
/install.sh
Если все сделано правильно, процесс завершится без ошибок.
Вывод должен быть примерно таким: Стандартный вывод
# .
/install.sh Checking minimum requirements. Creating volumes for persistent storage. Created sentry-data. .
env already exists, skipped creation. Building and tagging Docker images. smtp uses an image, skipping memcached uses an image, skipping redis uses an image, skipping Building web Step 1/2 : ARG SENTRY_IMAGE Step 2/2 : FROM ${SENTRY_IMAGE:- sentry:9.1.2}-onbuild # Executing 4 build triggers ---> Running in 6c97f9fcaf63 DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support Removing intermediate container 6c97f9fcaf63 ---> Running in 9e0f65ee3af6 Removing intermediate container 9e0f65ee3af6 ---> Running in 09754c44319c Removing intermediate container 09754c44319c ---> a12fa31c2675 Successfully built a12fa31c2675 Successfully tagged onpremise_web:latest Building cron Step 1/2 : ARG SENTRY_IMAGE Step 2/2 : FROM ${SENTRY_IMAGE:- sentry:9.1.2}-onbuild # Executing 4 build triggers ---> Using cache ---> Using cache ---> Using cache ---> Using cache ---> a12fa31c2675 Successfully built a12fa31c2675 Successfully tagged onpremise_cron:latest Building worker Step 1/2 : ARG SENTRY_IMAGE Step 2/2 : FROM ${SENTRY_IMAGE:- sentry:9.1.2}-onbuild # Executing 4 build triggers ---> Using cache ---> Using cache ---> Using cache ---> Using cache ---> a12fa31c2675 Successfully built a12fa31c2675 Successfully tagged onpremise_worker:latest Docker images built. Generating secret key. Secret key written to .
env Setting up database. Starting onpremise_smtp_1 .
done Starting onpremise_redis_1 .
done Starting onpremise_memcached_1 .
done 21:35:07 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured. 21:35:10 [INFO] sentry.plugins.github: apps-not-configured Syncing. Creating tables .
Installing custom SQL .
Installing indexes .
Installed 0 object(s) from 0 fixture(s) Migrating. Running migrations for sentry: - Migrating forwards to 0472_auto__add_field_sentryapp_author. > sentry:0424_auto__add_field_integration_status > sentry:0425_auto__add_index_pullrequest_organization_id_merge_commit_sha > sentry:0425_remove_invalid_github_idps > sentry:0426_auto__add_sentryappinstallation__add_sentryapp__add_field_user_is_sent > sentry:0427_auto__add_eventattachment__add_unique_eventattachment_project_id_event > sentry:0428_auto__add_index_eventattachment_project_id_date_added > sentry:0429_auto__add_integrationexternalproject__add_unique_integrationexternalpr > sentry:0430_auto__add_field_organizationintegration_status > sentry:0431_auto__add_field_externalissue_metadata > sentry:0432_auto__add_field_relay_is_internal > sentry:0432_auto__add_index_userreport_date_added__add_index_eventattachment_date_ > sentry:0433_auto__add_field_relay_is_internal__add_field_userip_country_code__add_ > sentry:0434_auto__add_discoversavedqueryproject__add_unique_discoversavedqueryproj > sentry:0435_auto__add_field_discoversavedquery_created_by > sentry:0436_rename_projectdsymfile_to_projectdebugfile > sentry:0437_auto__add_field_sentryapp_status > sentry:0438_auto__add_index_sentryapp_status__chg_field_sentryapp_proxy_user__chg_ > sentry:0439_auto__chg_field_sentryapp_owner > sentry:0440_auto__del_unique_projectdebugfile_project_debug_id__add_index_projectd > sentry:0441_auto__add_field_projectdebugfile_data > sentry:0442_auto__add_projectcficachefile__add_unique_projectcficachefile_project_ > sentry:0443_auto__add_field_organizationmember_token_expires_at > sentry:0443_auto__del_dsymapp__del_unique_dsymapp_project_platform_app_id__del_ver > sentry:0444_auto__add_sentryappavatar__add_field_sentryapp_redirect_url__add_field > sentry:0445_auto__add_promptsactivity__add_unique_promptsactivity_user_feature_org > sentry:0446_auto__add_index_promptsactivity_project_id > sentry:0447_auto__del_field_promptsactivity_organization__add_field_promptsactivit > sentry:0448_auto__add_field_sentryapp_is_alertable > sentry:0449_auto__chg_field_release_owner > sentry:0450_auto__del_grouphashtombstone__del_unique_grouphashtombstone_project_ha > sentry:0451_auto__del_field_projectbookmark_project_id__add_field_projectbookmark_ > sentry:0452_auto__add_field_sentryapp_events > sentry:0452_auto__del_field_releaseenvironment_organization_id__del_field_releasee > sentry:0453_auto__add_index_releasefile_release_name > sentry:0454_resolve_duplicate_0452 > sentry:0455_auto__add_field_groupenvironment_first_seen > sentry:0456_auto__add_dashboard__add_unique_dashboard_organization_title__add_widg > sentry:0457_auto__add_field_savedsearch_is_global__chg_field_savedsearch_project__ > sentry:0457_auto__add_monitorcheckin__add_monitor__add_index_monitor_type_next_che > sentry:0458_global_searches_data_migration Saved Searchs: 100% |########################################################################################################################################################################| Time: 0:00:07 > sentry:0459_global_searches_unique_constraint > sentry:0460_auto__add_field_servicehook_organization_id > sentry:0461_event_attachment_indexes > sentry:0462_auto__add_servicehookproject > sentry:0462_releaseenvironment_project_id > sentry:0463_backfill_service_hook_project > sentry:0464_auto__add_sentryappcomponent__add_field_sentryapp_schema > sentry:0464_groupenvironment_foreignkeys > sentry:0465_sync > sentry:0466_auto__add_platformexternalissue__add_unique_platformexternalissue_grou > sentry:0467_backfill_integration_status > sentry:0468_auto__add_field_projectdebugfile_code_id__add_index_projectdebugfile_p > sentry:0468_recent_search > sentry:0469_fix_state > sentry:0470_org_saved_search > sentry:0471_global_saved_search_types > sentry:0472_auto__add_field_sentryapp_author The following content types are stale and need to be deleted: sentry | dsymobject sentry | dsymapp sentry | useridentity sentry | dsymsymbol sentry | dsymsdk sentry | globaldsymfile sentry | versiondsymfile sentry | projectdsymfile sentry | grouphashtombstone sentry | minidumpfile sentry | dsymbundle Any objects related to these content types by a foreign key will also be deleted. Are you sure you want to delete these content types? If you're unsure, answer 'no'.
Type 'yes' to continue, or 'no' to cancel: yes
Running migrations for sentry.nodestore:
- Nothing to migrate.
Running migrations for sentry.search:
- Nothing to migrate.
Running migrations for social_auth:
- Nothing to migrate.
Running migrations for sentry.tagstore:
- Nothing to migrate.
Running migrations for sentry_plugins.hipchat_ac:
- Nothing to migrate.
Running migrations for sentry_plugins.jira_ac:
- Nothing to migrate.
Synced:
> django.contrib.admin
> django.contrib.auth
> django.contrib.contenttypes
> django.contrib.messages
> django.contrib.sessions
> django.contrib.sites
> django.contrib.staticfiles
> crispy_forms
> debug_toolbar
> rest_framework
> sentry.plugins.sentry_interface_types
> sentry.plugins.sentry_mail
> sentry.plugins.sentry_urls
> sentry.plugins.sentry_useragents
> sentry.plugins.sentry_webhooks
> sudo
> south
Migrated:
- sentry
- sentry.nodestore
- sentry.search
- social_auth
- sentry.tagstore
- sentry_plugins.hipchat_ac
- sentry_plugins.jira_ac
Creating missing DSNs
Correcting Group.num_comments counter
Cleaning up.
----------------
You're all done! Run the following command get Sentry running:
docker-compose up -d
Всё, версия 9.1.2 установлена.
Выводы.
Это общеизвестные истины.
Не торопитесь, делайте резервные копии и проверяйте все несколько раз.
Спасибо за внимание! Теги: #DevOps #тестирование веб-сервисов #резервное копирование #sentry
-
Основные Преимущества Zextras Powerstore
19 Oct, 24 -
30-Летие Безудержной Нестабильности
19 Oct, 24 -
Кодирование Без Кода
19 Oct, 24 -
Прототипирование С Помощью Wireframesketcher
19 Oct, 24
