Tuesday, May 3, 2022, 11:31am PT
**Please see final updates in red towards the bottom of the article.**
Our apologies for the trouble creating and managing food related items and data sync right now. We saw this issue arise around 10am PT on Friday, April 22nd. Current symptoms include, but are not limited to the following:
- inability to create a new food item or edit an existing food item in our database
- inability to create or edit a My Meal, My Food, or Recipe
- inability to add to diary via the Quick Add feature
- inability to complete diary
- inability to create or edit Recipes
- recipes, foods, meals, and other account data not visible in app after a reinstall
- inability to complete diary
- inability to login - these members can still access their account via the website. Work completed Wed 4/27 has enabled many members to start logging in again.
- iOS - some members receiving a server error. this seems to manifest after experiencing a crash when editing/creating a recipe, or after one logs out or reinstalls the app. We released a fix to prevent users from getting into this state, but accounts already in this corrupt state prior to the fix who haven't yet logged out/reinstalled may still experience this issue.
- Android - some members receiving a "Registration of new MyFitnessPal accounts is not available in your location" error when trying to log in, and their account may show as a local account.
- syncing trouble - members may experience issues with data sync. Work completed Wed 4/27 has improved sync for some members, though we are still working to resolve this for all members.
- If members create new food, a meal, or use quick add on their account, these newly created items will be stored locally on their device, and will not sync to other platforms (like the website), until this issue is resolved.
- If you do not see your data on the website, whether it be newly created/edite foods/recipes/etc, or food logged to your diary, it means that this data has not synced to our server, and could potentially be lost if the app is reinstalled or if you have logged out of the app.
- You may log any of the data you don't see on the website, but do see in the app, directly to the website to ensure that this data syncs to our server. Two caveats with this: 1. there are more features impacted on the website (see web section above), so you may still be unable to log certain things to your account, and 2. If/when your mobile data does sync, it may result in duplicated data in your account.
- while the services that support syncing data to partners are not affected, the sync issues affecting the app are affecting data syncing from the app to our backend servers. This in turn may result in data not reaching linked partners (e.g. Fitbit, Garmin, etc) & vice versa.
There is no workaround at this time to get these food items created or edited. We do not recommend resintalling the app at this time, as it will not resolve the issues mentioned above, and may result in additional issues due to the trouble with sync.
Our team identified the cause Friday afternoon and worked through out Friday and Saturday to get the necessary migration work in place to address to start addressing the root cause. This migration has made a lot of progress throughout Sunday and Monday, and finished Tuesday afternoon.
Our team spent Tuesday updating our services to support the database changes that have been made after the completion of the migration. We completed what we could that would have no additional user impact during the day when the majority of our members are online and logging. Tuesday evening, from roughly 8pm-2:15am PT Wed morning, we upgraded the rest of the services, which resulted in MFP being mostly unavailable both in the apps and on the website during this window.
Now that we finished the updates to our services, the rest of Wednesday was dedicated to working on and deploying code changes to start resolving the above bulleted list of known symptoms. Over the course of Wednesday, our team cleaned up a lot of corrupt data and rebuilt databases. This data cleanup has resolved the inability to log into the mobile apps for many users. Additionally some users (not all) have reported improvement in relation to data sync as well.
Over the course of Thursday we prepared our databases for the maintenance scheduled for 7-9pm PT Thursday evening. We had planned for two hours of downtime, and finished about an hour ahead of schedule. It was expected for members to still be experiencing symptoms after this maintenance.
Our team got an early start Friday to complete the final steps in our resolution plan and was able to get feature functionality restored to our members as of 7:15am PT. We ran through all of the bulleted list of symptoms and are no longer able to reproduce this in our accounts.
Our engineering team continued to monitor things for stability on the backend throughout the weekend, and through the entirety of Monday. All systems and reporting indicates that the changes made have remained successful.
Our support team has also continued to monitor our support queues for reports of ongoing issues. While the majority of symptoms have resolved, we do have a small subset of members on iOS that seem to be having difficulty logging into the app and our engineering team is working closely alongside our support team to determine if it's related to this incident and needs to be fixed by our backend team, or if it's unrelated and needs to be investigated by our iOS team. Please reach out to our support team here, if you need assistance with this.
Any other lingering issues having been reported have been on a one off basis, and nothing appears to be an ongoing issue affecting a large group of our members.
A few items to note:
- It is not expected that any historical data is missing or affected, however it is possible that some food related data logged on mobile during the incident may not be available (depending on the circumstances of the specific account). Due to the nature of the sync issues, if data was logged in the app during the active time of this incident (last Friday to this morning), and stored locally on a device, and the app was reinstalled, or the account was logged out of, then this data may no longer be in the app and able to sync.
- If you are still having trouble with partner sync after 7:15am PT / 9:15am CT, we recommend unlinking the partner and relinking the partner in order to have a fresh connection and to prompt sync to occur. Only data from the date of relinking and forward is expected to sync; Unsynced data from previous days will not sync.
- If you need to correct your login streak, please see this article.
All of that said, if you are still experiencing issues, please reach out to our support team here, so that we can triage these with you. Click on the chat bubble icon in the lower right corner, follow the prompts, and then click the "contact support" option to send our support team an email. Please be as specific as possible as to the issue you are experiencing, as the more details initially provided, the faster we can investigate.