Job Description

As a member of NBCUniversal’s Production Software Engineering (PSE) team, the primary purpose of the Senior Production & Technology Support Engineer is to provide support for NBCUniversal’s virtualized production infrastructure and applications critical to on-air production, custom built software, and infrastructure. This role will be responsible for providing L1 through L3 support (depending on which application) for our 24x7x365 applications used in newsrooms, studios, control rooms, and the field across NBCU.

This role encompasses a focus on virtualized production infrastructure and production software critical on-air applications, requiring hands-on 24/7/365 support for application infrastructure and collaborating with cross-functional teams and 3rd party vendors to enhance and optimize technology for a major television network. The position requires weekday, weekend, and holiday shifts, as live on-air operations run continuously, with weekends and holidays treated as standard workdays

Responsibilities:

• Provide Support to Virtualized Productions and Production Software Applications * Support live programming for NBCU Local, MSNBC, CNBC, TODAY, Nightly News, Dateline, Telemundo, and other NBCU shows and networks * Responsible for the support of live production operations within the facility for on-air production * Respond to and resolve reported issues from users and L1 using best practices to ensure production reliability. * Communicate and escalate issues to L3 support & engineering managers as needed. * Passionate about the production process, excited by new technologies and opportunities and ready to help develop and improve the existing process and infrastructure. * Self-directed ownership of tasks and assignments * Able to provide adequate status updates to senior and executive leadership communicating both problems and solutions to technical and non-technical personnel alike to help develop new processes, technologies, and techniques, which will help to improve the funning of live on air shows and productions
• Validation of Software Deployments: * Assist and collaborate with the development team in validating new platform software releases. * Identify bugs and workflow gaps through operational testing.
• Documentation and Workflow Development: * Identify and resolve gaps in documentation. * Create Standard Operating Procedure (SOP) documents for efficient issue resolution for use by technical operations and other support staff to resolve issues efficiently. Regulates, documents and implements Standard Operating Procedures (SOPs) * Create new SOPs and operational guidelines and train others in these SOPs, issue resolution, and recovery * Researching and documenting production issues or potential on-air discrepancies- providing detailed write-ups of occurrences resulting in potential outages or degraded performances
• Application Support: * Monitor incoming user issues (tickets and/or emails) and provide hands-on application support and troubleshooting for 24x7x365 virtualized productions and production software applications, escalating to the appropriate group when required staying in alignment with set SLAs * Collaborate with cross-functional teams to define support SOPs and create /define playbook documentation. Including 3rd party vendors * Performs routine system updates, security patching, collaboration with various tech teams regarding planned maintenance, and provides on call 24x7x365 on-call coverage * 24x7x365 availability for production outages, emergencies, deployments, and escalations
• Collaboration and Coordination: * Work closely with leaders to coordinate maintenance and build activities around production schedules. * Collaborate with various teams, including Scrum Leads, Software Engineers, QA Engineers, Product Manager, Project Managers, Producers, Operational Partners (Contractors and/or Staff) and third-party vendors. * Works with product /business engagement teams to learn business objectives, development teams to plan support needs, QA to understand test strategy, and SRE on environments, deployments, and regular maintenance * Participates in Scrums, demos, and other Agile ceremonies and ensures accurate and timely status updates to the team * Hands-on support and documentation for all on-prem & cloud (AWS) environments (Servers, Storage, Databases, Networking, Security, etc.) * Observes and responds to all application, infrastructure, and services monitoring alerts and dashboards * Recognizes and identifies potential operational impacts of application or enterprise change controls which could affect our applications and customers * Helps improve performance, scalability, and reliability * Performs routine system updates, security patching, collaboration with various tech teams regarding planned maintenance, and provides on call 24x7x365 on-call coverage * 24x7x365 availability for production outages, emergencies, deployments, and escalations
• Outage Response and Root Cause Analysis: * Own outage response, management, resolution, and root cause analysis. Incident, Service, Knowledge & Change Management * Establish subject matter expertise for all applications and workflows. * 24x7x365 availability for production outages, emergencies, deployments, and escalations
• Metrics and Continuous Improvement: * Implements gathering and utilization of metrics to measure accomplishments and monitor progress, ensuring SLAs are met and customers are satisfied with their support experience to measure accomplishments and monitor progress. * Contribute to process improvement ideas and support innovation.

Qualifications

Basic Requirements:

• 5+ years of experience supporting large-scale, mission-critical applications.
• Proficient in agile, Github, and ITSM tools like Auto Now Service Now.
• Experience troubleshooting a broad range of applications, desktop & mobile operating systems, browsers, etc.
• Must have experience working directly with software development teams in a Continuous Delivery and SDLC environment
• Knowledge of a broad range of enterprise technologies
• Must have ability to review and interpret application, infrastructure, and service logs
• Must have basic knowledge of AWS services (IAM, EC2, RDS, S3, etc.)
• Must have knowledge across cloud providers (AWS, Azure, GCP) a plus
• Knowledge of NoSQL data stores (MarkLogic, MongoDB, Cassandra, DynamoDB, Couchbase, PostgreSQL, etc.) a plus
• Knowledge on all broadcasting equipment and included but not limited to Production Control Switchers, Broadcast Routers, CG, DV, Audio and Video Mixers, RTS, Riddle, Clearcom, EVS, Icon Stations, Ross XPression, Tessera / Graphite / Carbonite, Dreamcathcer, Media Asset Management (NAM, DAM), Encoders/ Decoders, Rasterizers, Evertz AIR Cleaners, Motion, FTP client and Omneon Spectrum, Versio, Vue Routers, Nexio, Utah routers, LiveU, TVU, Evertz Magnum UI, EXQ and EXE routers.
• Strong Knowledge of automation and IT systems. (Automations: ITX, Harris, Morpheus and Mediator X)
• Well-versed on Broadcast operations and procedural requirements to confirm accuracy and on-air readiness of scheduled material by checking the IRD, Traffic log and automation playlists. Creation of VOD for streaming platforms.
• Troubleshooting Audio, Video and other technical issues that may arise in the uplink/downlink signal path and be able to recover quickly and effectively.
• Optimizing workflows related to all broadcast systems including linear and IP fabric for video infrastructures, virtualized Windows and Linux environments, Encode / Decode platforms (software / hardware), newsgathering tools, Non-Linear Editing, Distribution/ Transmission systems
• Fluent in Microsoft Office applications, including Outlook, Teams, Word, Excel, etc.
• Must be willing to work evenings, overnights, weekends, holidays with little or/no notice in support of 24x7x365 critical system availability
• Must be willing to travel on an as-needed basis
• Familiarity with AWS services and cloud providers is a plus.
• Experience within a News or Television Production Environment / Media & Entertainment a must

Desired Characteristics:

• Responsive and organized, with a high tolerance for stressful situations.
• Inclusive and collaborative approach with strong interpersonal and communication skills.
• Detail-oriented and resourceful with a drive to learn.
• Strong problem-solving and written communication skills.
• Flexibility to work according to production software engineering schedules, including evenings, overnights, weekends, and holidays.
• Passion for monitoring, metrics, and continuous improvement.
• Extremely responsive and organized, letting nothing fall through the cracks
• Solves problems quickly and automates processes for the future
• Knows when to place a call to a user rather than continuing a long email thread or chat
• Ability to take responsibility, jump in and get things done, and be self-directed with limited supervision.
• Ability to recognize priorities and pivot activities accordingly. Ability to focus on deadlines and deliverables. Ability to work and participate in a team with an open and collaborative style of communication
• Demonstrated self-composure, high tolerance for stressful situations, and unquestionable integrity under pressure.
• Strong interpersonal and communication skills with ability to effectively speak with people at all technical levels, from CTOs and CIOs to users and business stakeholders
• Highly customer focused and very responsive. Listens, understands and acts
• An approach that is inclusive of other viewpoints and perspectives; confidence to make tough decisions about difficult issues with limited data
• Big Picture thinker
• Demonstrated ability to work independently as well as part of a cross-functional team
• Takes accountability for action items and requests even those requiring actions outside of their direct control, following up and leaving nothing to chance

Additional Requirements:

• Fully Remote: This position has been designated as fully remote, meaning that the position is expected to contribute from a non-NBCUniversal worksite, most commonly an employee’s residence.

This position is eligible for company sponsored benefits, including medical, dental and vision insurance, 401(k), paid leave, tuition reimbursement, and a variety of other discounts and perks. Learn more about the benefits offered by NBCUniversal by visiting the Benefits page of the Careers website.

Salary range: $100,000 - $130,000

We are accepting applications for this position on an ongoing basis.

Additional Information

As part of our selection process, external candidates may be required to attend an in-person interview with an NBCUniversal employee at one of our locations prior to a hiring decision. NBCUniversal's policy is to provide equal employment opportunities to all applicants and employees without regard to race, color, religion, creed, gender, gender identity or expression, age, national origin or ancestry, citizenship, disability, sexual orientation, marital status, pregnancy, veteran status, membership in the uniformed services, genetic information, or any other basis protected by applicable law.

If you are a qualified individual with a disability or a disabled veteran and require support throughout the application and/or recruitment process as a result of your disability, you have the right to request a reasonable accommodation. You can submit your request to AccessibilitySupport@nbcuni.com.

For LA County and City Residents Only: NBCUniversal will consider for employment
qualified applicants with criminal histories, or arrest or conviction records, in a manner
consistent with relevant legal requirements, including the City of Los Angeles' Fair Chance
Initiative For Hiring Ordinance, the Los Angeles' County Fair Chance Ordinance for Employers, and the California Fair Chance Act, where applicable.