TestEngineering/Services/LoopTestEnvironments: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
 
(24 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== Loop Server Production Environments ==
<font size=6><span style="color:#FF0000">This wiki has moved!</span><br /><br />------> [[Loop/Test]]</font>
* Loop-Server: https://loop.services.mozilla.com
* Loop-Client: https://call.mozilla.com
* MSISDN Gateway: https://msisdn.services.mozilla.com
 
* Monitoring
** See below: https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Monitoring_the_Production_Environment
 
* QA Access: not allowed
 
* Pointing Fx Nightly to Production
** NOTE: Fx Nightly is already set to point to Production by default
 
* Quick Verifications
Loop-Server
In the browser: https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server
curl https://loop.services.mozilla.com
Note: the default value of "fakeTokBox" is true for the mock server, false for the live server
curl -I https://loop.services.mozilla.com
 
Loop-Client
In the browser: https://call.mozilla.com
Welcome to the Loop web client.
Browser: https://call.mozilla.com/config.js
var loop = loop || {};
loop.config = {serverUrl: 'https://loop.services.mozilla.com'};
Browser: Browser: https://call.mozilla.com/VERSION.txt
curl https://call.mozilla.com/
returns a page of HTML
curl -I https://call.mozilla.com/
 
MSISDN-Gateway
Browser: https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN 
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}
curl https://msisdn.services.mozilla.com
{"name":"mozilla-msisdn-gateway","description":"The Mozilla MSISDN
Gateway","version":"0.3.2","homepage":"https://github.com/mozilla-services/msisdn-gateway"
,"endpoint":"https://msisdn.services.mozilla.com"}
curl -I https://msisdn.services.mozilla.com
 
== Loop Server Stage Environment ==
* URLs
** Loop-Server: https://loop.stage.mozaws.net
** Mock (load): https://loop-delayed-response.stage.mozaws.net (see below)
** Third-Party (load): TBD
 
** loop-redis-stage - need the details
 
** Loop-Client: https://call.stage.mozaws.net
 
** MSISDN: https://msisdn.stage.mozaws.net (see below)
** Mock (load): http://omxen.dev.mozaws.net
 
** Sentry: https://sentry.shared.us-east-1.stage.mozaws.net (see below)
 
== Loop Server Stage Details ==
* URL
** Loop-Server: https://loop.stage.mozaws.net
 
* Versions
** rpm -qa | grep loop
** Example:
*** loop-server-svcops 0.10.0-1 x86_64 20905864
*** puppet-config-loop 20140811172607-1 x86_64 14277
 
* VERSION.txt
** ./data/loop-server/VERSION.txt
 
* AWS
** CF stack: loop-server-stage (or similar name)
** ELB: loop-server-st-ELB-<some value>
** One instance: look for "loop-server-stage"
 
* OS
** Running SciLinux 6.4
 
* Files
** /data/loop-server
*** especially /data/loop-server/config/settings.json
** /data/hekad
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /etc/circus.ini
** /etc/circus.d
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/puppet/*
*** especially /etc/puppet/yaml/app
*** /etc/puppet/yaml/app/loop_server.dev.yaml
*** /etc/puppet/yaml/app/loop_server.prod.yaml
*** /etc/puppet/yaml/app/loop_server.stage.yaml
*** /etc/puppet/yaml/app/loop_server.yaml
** /var/log/*
 
* Processes
** node loop/index.js
** stackdriver
** circusd
** heka.d
** nginx
 
* Logs
** /var/log/circus.log
** /var/log/hekad/loop.stdout.log
** /var/log/hekad/loop.stderr.log
** /media/ephemeral0/circus/loop_server/loop_server.err.log
** /media/ephemeral0/circus/loop_server/loop_server.eout.log
** /media/ephemeral0/nginx/logs/default.access.log (not in use)
** /media/ephemeral0/nginx/logs/default.error.log (not in use)
** /media/ephemeral0/nginx/logs/loop_server.access.log
** /media/ephemeral0/nginx/logs/loop_server.error.log
 
* Setting Stage configuration:
** Toggling the config setting to use either a mock server or a third-party server
** File: /data/loop-server/config/settings.json
 
* Quick Verifications
curl https://loop.stage.mozaws.net
curl -I https://loop.stage.mozaws.net
In the browser: https://loop.stage.mozaws.net/
 
== Loop Client and Server Configurations ==
* Live Server vs. Mock Server
** By default, the Loop-Server in Stage is configured to talk to our mock server:
*** https://loop-delayed-response.stage.mozaws.net/
** Default load testing will hit the mock server.
** To hit the partner test servers, the following configuration file will need to be updated by OPs:
*** /data/loop-server/config/settings.json
** Talk to OPs to toggle that configuration file and restart the Loop-Server in Stage.
** REFs:
*** https://wiki.mozilla.org/QA/Services/LoopTestEnvironments#Loop_Mock_Server_Stage_Details
 
* Yaml Files
** /etc/puppet/yaml/app
*** loop_client.dev.yaml
*** loop_client.prod.yaml
*** loop_client.stage.yaml
*** loop_client.yaml
*** loop_server.dev.yaml
*** loop_server.prod.yaml
*** loop_server.stage.yaml
*** loop_server.yaml
 
== Loop Client Stage Details ==
* URL
** Loop-Client: https://call.stage.mozaws.net
 
* Versions
** rpm -qa | grep loop
 
* VERSION.txt
** /data/loop-client/content/VERSION.txt
 
* AWS
** CF Stack: look for loop-client-*
** ELB: loop-client-st-ELB-<some value>
** One instance: look for "loop-client app server"
 
* OS
** Scientific Linux release 6.4
 
* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/loop-client
*** Note: this does not contain any app-specific config file
** /data/hekad
** /etc/heka.d
** /etc/puppet/yaml
*** especially the following files
*** /etc/puppet/yaml/app/loop_client.dev.yaml
*** /etc/puppet/yaml/app/loop_client.prod.yaml
*** /etc/puppet/yaml/app/loop_client.stage.yaml
*** /etc/puppet/yaml/app/loop_client.yaml
 
* Processes
** look for stackdriver, nginx, python, circus, heka
 
* Logs
** /var/log/circus.log
** /var/log/hekad
** /media/ephemeral0/nginx/logs
 
* Heka
** /data/hekad
** /etc/heka.d
 
* Quick Verifications
In the browser: https://call.stage.mozaws.net
 
Check the version: https://call.stage.mozaws.net/VERSION.txt
curl https://call.stage.mozaws.net
curl -I https://call.stage.mozaws.net
Check that Loop-Client Stage is talking to Loop-Server Stage
In the browser: https://call.stage.mozaws.net/config.js
should return something similar to this
var loop = loop || {};
loop.config = {serverUrl: 'https://https://loop.stage.mozaws.net'};
 
== Loop Mock Server Stage Details ==
** NOTE: This server is used for load testing whenever we do not want to hit the third-party servers. A configuration change must be made in the Loop Server in order to make this work.
 
* URL
** Mock: https://loop-delayed-response.stage.mozaws.net
 
* AWS
** CF stack: loop-delayed-response (or similar)
** ELB: loop-delayed-r-ELB-<some value>
** R53AliasRecord: loop-delayed-response.stage.mozaws.net
** Instances: 1 m1.small, search for "delayed HTTP server"
 
* OS
** Scientific Linux release 6.4
 
* Files
** /data/delayed
** /data/delayed/delayed-master
** /etc/circus.d/
** /etc/circus.d/loopserver.ini
** /etc/circus.d/hekad-loop.ini
** /etc/circus.ini
 
* Processes
** root /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/
** nobody node index.js
 
* Logs
** /var/log/circus.log
 
* Quick Verifications
In the browser: https://loop-delayed-response.stage.mozaws.net
ok
 
* Access
** Log in as "ec2-user"
 
== MSISDN Gateway Server Stage Details ==
* URL
** MSISDN Gateway: https://msisdn.stage.mozaws.net
 
* Versions
** rpm -qa
** Current:
*** msisdn-gateway-svcops 0.4.2-0snap201408110138git80ea3b x86_64 52573829
*** puppet-config-msisdn 20140811185304-1 x86_64 12723
 
* AWS
** CF Stack: msisdn-gateway-stage (or similar)
** ELB: msisdn-gateway-ELB-<SOME VALUE>
** Currently one m3.medium instance
 
* OS
** Scientific Linux release 6.4 (Carbon)
 
* Files
** /opt/aws
** /opt/ec2
** /opt/openresty
** /opt/rh
** /opt/stackdriver
** /data/hekad
** /data/msisdn-gateway
 
* Processes
** Look for stackdriver, python, circus, resty, nginx, node, heka
 
* Logs
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_err.log
** /media/ephemeral0/msisdn-gateway/msisdn-gateway_out.log
** /media/ephemeral0/nginx/logsdefault.access.log (not in use)
** /media/ephemeral0/nginx/logsdefault.error.log (not in use)
** /media/ephemeral0/nginx/logsmsisdn-gateway.access.log
** /media/ephemeral0/nginx/logsmsisdn-gateway.error.log
** /var/log/circus.log
** /var/log/hekad/msisdn_gateway.stderr.log
** /var/log/hekad/msisdn_gateway.stdout.log
 
* Configuration File
** /data/msisdn-gateway/config/production.json
 
* Quick Verifications
In the browser: https://msisdn.stage.mozaws.net
https://msisdn.stage.mozaws.net/__heartbeat__
returns {"storage":true}
curl https://msisdn.stage.mozaws.net
curl -I https://msisdn.stage.mozaws.net
 
== MSISDN Mock Server Stage Details ==
* URL: http://omxen.dev.mozaws.net
* Endpoint: http://omxen.dev.mozaws.net/send
 
* AWS
** This is one m3.medium
** Look for omxen.dev.mozaws.net
 
*Access: SSH
 
* OS
** Amazon Linux AMI release 2014.03
 
* Versions
** TBD
 
* Processes
** Look for nginx, python, omxen
 
* Files
** /home/ec2-user/omxen
** /opt/aws
 
* Logs
** /var/log/nginx
** Use: sudo cat nginx/*.log, etc
 
* Restarting
** TBD
 
* Quick Verifications
In the browser, http://omxen.dev.mozaws.net/
OMXEN SMS GATEWAY
curl omxen.dev.mozaws.net
OMXEN SMS GATEWAY
curl -I omxen.dev.mozaws.net
 
== Sentry Server Stage Details ==
* URL
** https://sentry.shared.us-east-1.stage.mozaws.net
 
* Versions
** rpm -qa | grep sentry
 
* AWS
** TBD
 
* OS
** Scientific Linux release 6.3
 
* Files
** /opt/sentry
 
* Processes
** root    /usr/bin/python /usr/bin/circusd --log-output /var/log/circus.log --pidfile /var/run/                                                       
** apache    /opt/sentry/bin/python /opt/sentry/bin/sentry --config=/etc/sentry.d/sentry.shared.us-east-1.stage.mozaws.net.py start ud
 
* Logs
** /var/log
** nothing in /media/ephemeral0
 
* Heka
** TBD
 
* Quick Verifications
** For now, just verify the service is running on the host: ps aux | grep sentry
 
== QA Access via a Bastion Host ==
* SSH with AWS keys to the Stage bastion host in US East 1. From there SSH directly into any instance using your LDAP name. For the mock server, log in as "ec2-user".
 
== Monitoring the Stage Environment ==
* Sentry: https://sentry.shared.us-east-1.stage.mozaws.net
** Getting access to Sentry Stage - talk to the OPs team
 
* Stackdriver
** https://app.stackdriver.com/
** None at this time. See https://bugzilla.mozilla.org/show_bug.cgi?id=1063134
 
* Graphite
** https://graphite.shared.us-east-1.stage.mozaws.net
*** Graphite > stats > loop-activated-users
*** Graphite > stats > loop-call-urls
*** User Graphs > not currently defined
 
* Kibana:
** https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
** https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** HTTP Request Times - TBD
 
* Heka
* https://heka.shared.us-east-1.stage.mozaws.net/
* (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
** PlugIns - Filters
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
** PlugIns - Outputs
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopNginxErrorFileOutput
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/outputs/LoopServerFileOutput
** PlugIns - Encoders
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
*** https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
** Sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
*** https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf
 
== Monitoring the Production Environment ==
* Sentry: http://sentry.shared.us-west-2.prod.mozaws.net
** Getting access to Sentry Prod - talk to the OPs team
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
** http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/
 
* Kibana:
** Main: https://kibana.shared.us-west-2.prod.mozaws.net
** https://kibana.shared.us-west-2.prod.mozaws.net/index.html#/dashboard/file/loop_http_status.json
** https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/QeeHeG7MQoGv4HdKzANCEw
 
* Heka:
** Main: https://heka.shared.us-west-2.prod.mozaws.net/#
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
** https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus
 
* Loop-Server Dashboard
** https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard
 
* Graphite: https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)
 
* Firefox Input Dashboard: https://input.mozilla.org/en-US/?product=Loop
 
* Pingdom:
** Loop-Client: http://stats.pingdom.com/20dar76w4hmv/1304575
** Loop-Server: http://stats.pingdom.com/20dar76w4hmv/1304565
 
== Loads V1 Services Cluster Environment ==
* For details on the Loads V1 cluster, see the following link:
** https://wiki.mozilla.org/QA/Services/LoadsToolsAndTesting1
 
== Dev Environment ==
* URLs
** Server: https://loop-dev.stage.mozaws.net/
** Client: http://loop-webapp.dev.mozaws.net/
** Server: https://msisdn-dev.stage.mozaws.net/
* Versions
** TBD
 
* AWS
** CF Stack: ulfr-devstack-loop
** ELB: ulfr-devstack-ELB-<some value>
** One instance: search for "loop app server (ulfr-devstack-loop)"
 
** CF Stack: bwong-msisdn-dev
** ELB: bwong-msisdn-d-ELB-<some value>
** One instance: search for "msisdn_gateway server (bwong-msisdn-dev)"
 
* OS
** TBD
 
* Files
** TBD
 
* Processes
** TBD
 
* Logs
** TBD
 
* Heka
** TBD
 
* Quick Verifications
http://loop.dev.mozaws.net
Returns
{
  "name": "mozilla-loop-server",
  "description": "The Mozilla Loop (WebRTC App) server",
  "version": "0.7.0",
  "homepage": "https://github.com/mozilla-services/loop-server/",
  "endpoint": "http://loop.dev.mozaws.net",
  "fakeTokBox": false
}
or similar
http://loop.dev.mozaws.net/__heartbeat__
Returns
  {
  "storage": true,
  "provider": false
}
or similar
 
http://loop-webapp.dev.mozaws.net
Returns
    Loop
    Welcome to the Loop web client.
http://loop-webapp.dev.mozaws.net/__heartbeat__
Returns 404
 
http://msisdn.dev.mozaws.net
Returns
{
  "name": "mozilla-msisdn-gateway",
  "description": "The Mozilla MSISDN Gateway",
  "version": "0.3.0-DEV",
  "homepage": "https://github.com/mozilla-services/msisdn-gateway/",
  "endpoint": "https://msisdn.dev.mozaws.net"
}
or similar
http://msisdn.dev.mozaws.net/__heartbeat__
Returns
{
  "storage": true
}
or similar
 
https://msisdn-dev.stage.mozaws.net
Returns
{
  "name": "mozilla-msisdn-gateway",
  "description": "The Mozilla MSISDN Gateway",
  "version": "0.3.0-DEV",
  "homepage": "https://github.com/mozilla-services/msisdn-gateway/",
  "endpoint": "https://msisdn-dev.stage.mozaws.net"
}
or similar
https://msisdn-dev.stage.mozaws.net/__heartbeat__
Returns
{
  "storage": true
}
or similar
 
== References ==
* Main Sites
** https://wiki.mozilla.org/CloudServices/Loop
** https://wiki.mozilla.org/CloudServices/Loop/Server
** https://wiki.mozilla.org/Services/Loop
** https://wiki.mozilla.org/Loop
** https://wiki.mozilla.org/Loop/Architecture
** https://wiki.mozilla.org/Loop/Architecture/ID
** https://wiki.mozilla.org/Loop/Telemetry
** Loop MVP Features: https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AlJmiyFngeSwdGJEVnItQ2NVbGx6NXdjbVBUYmIxSGc#gid=0
** https://people.mozilla.org/~dhenein/labs/loop-docs/
** https://people.mozilla.org/~dhenein/labs/loop-mvp-spec/
** https://wiki.mozilla.org/Loop/Architecture/MVP
 
* Documentation
** https://docs.services.mozilla.com/loop/index.html
** https://docs.services.mozilla.com/loop/apis.html
 
* GitHub Repos
** https://github.com/orgs/mozilla-services/teams/loop
** https://github.com/mozilla-services/loop-server
** https://github.com/mozilla/loop-client
** https://github.com/mozilla/loop-client/releases
** https://github.com/mozilla-services/msisdn-gateway
** https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md
** https://github.com/mozilla/videur
** https://github.com/mozilla/fxa-auth-server/wiki/API-extensions-for-supporting-MSISDN-verification-in-FxA-auth-server
** https://github.com/mozilla/gecko-projects/tree/elm/browser/components/loop/standalone
** https://github.com/mozilla-services/svcops
** https://github.com/mozilla-services/puppet-config
 
* Bugzilla
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Loop&component=Server
*** Other Products
*** Product: Loop
*** Component: General/Client/Server
 
** https://bugzilla.mozilla.org/enter_bug.cgi?product=Mozilla%20Services@component=MobileID
*** Client Software
*** Product: Mozilla Services
*** Component: MobileID
 
== OPs Mana and GitHub Pages ==
* NOTE: Talk to OPs for the links to Mana.
* Puppet Config: https://github.com/mozilla-services/puppet-config
* Cloud Formation: https://github.com/mozilla-services/svcops
* Loop-specific: TBD

Latest revision as of 20:04, 26 August 2016

This wiki has moved!

------> Loop/Test

Retrieved from "https://wiki.mozilla.org/index.php?title=TestEngineering/Services/LoopTestEnvironments&oldid=1145595"