Skip to:
Content
Pages
Categories
Search
Top
Bottom

breaking down large sql queries to prevent long processes being killed


  • jaykdoe
    Participant

    @jaykdoe

    For starters, BuddyPress has been a crucial plugin for my website, because it is a social network that relies on community functions, but it has been such a resource intensive plugin that my website very rarely functions properly and even when it does it has ridiculous load times of 10-20 seconds, at which point it will often simply time out and throw an error.

    I recently moved my website over to WPengine hosting as it has a great reputation as a managed wordpress hosting platform, but that hasn’t really resolved any of my issues. If anything it’s made them more apparent and support has helped my identify that BuddyPress is the most often cuplrit of my issues.

    I recently had database error connections and according to the support team it has to do with BuddyPress not following best practices and some core functions utlizing larger sql queries that take so long that ultimately the host has to kill the process altogether.

    Here is the information provided to me by support
    “As your site is in a shared environment, we have protocols in place to ensure its functions are shared evenly between our customers. To that end, we have a long process killer that checks every 60 seconds for processes (including sql queries) that have been running for over a minute. This has the effect of terminating any process that has been running for between 60-119 seconds. If we did not do this, it would only take a small number of long processes to effectively disable an entire cluster of sites.
    Plugin developers should be aware of this, and ensure their code can batch their jobs into smaller chunks, instead of allowing long, server strangling processes to run, as we will not permit that.”

    For instance see below at the end of this post to see just one example of a larger query that took the site down. In short, my question is how can I batch these queries to function reliably. But looking at the broader picture, my question is, why can’t any hosting service reliably support a buddypress install to the point that my website doesn’t even function or load in under 20 seconds? I’ve seen many other people complain about how buddypress slows their website down to a crawl and most of the resolutions I have seen regard using caching, which I do in fact use with WPEngine. If WpEngine can’t reliably host a buddypress install, who can? Should it really be this difficult to reliably run a site on buddypress? Thanks in advance for any assistance and additional info anyone can provide.

    KILLED QUERY (44313 characters long generated in /nas/content/live/mysite/wp-content/plugins/buddypress/bp-members/bp-members-functions.php:762): SELECT COUNT(user_id) FROM wp_dt89y1h23p_bp_activity WHERE component = ‘members’ AND type = ‘last_activity’ AND user_id NOT IN (48,106,152,379,423,645,834,1039,1516,2162,2164,2635,2776,2843,2852,2856,2857,2885,3092,3093,3095,3096,3163,3168,3263,3265,3267,3268,3332,3347,3348,3350,3351,3352,3579,3580,3581,3582,3583,3584,3586,3646,3702,3718,3719,3721,3723,3725,3726,3729,3740,3742,3743,3747,3748,3749,3754,3755,3756,3757,3758,3760,3761,3763,3764,3765,3768,3769,3770,3771,3772,3773,3775,3776,3777,3832,3835,3836,3842,3843,3896,3898,3900,3902,3911,3965,4181,4496,6423,6666,6835,7126,7205,7329,7668,7830,8228,8313,9197,10286,10780,11172,11259,11744,12796,13908,14031,14554,15371,16295,16320,16381,16736,16871,16879,17954,18095,18167,18644,18715,18833,19104,20228,20256,20743,21006,21307,21384,21432,21434,21566,21567,21568,21569,21570,21571,21572,21573,21574,21576,21584,21595,21596,21600,21603,21604,21610,21612,21614,21615,21616,21618,21619,21620,21630,21632,21642,21646,21650,21652,21653,21663,21664,21665,21667,21668,21669,21676,21678,21679,21680,21690,21698,21699,21700,21701,21702,21704,21705,21706,21707,21713,21737,21738,21739,21740,21741,21742,21743,21744,21745,21746,21747,21748,21749,21750,21751,21752,21753,21754,21759,21761,21767,21768,21774,21778,21782,21789,21793,21798,21806,21809,21817,21821,21823,21827,21830,21831,21841,21847,21850,21853,21854,21864,21870,21871,21880,21881,21882,21892,21893,21895,21896,21897,21898,21900,21901,21905,21908,21910,21911,21913,21917,21918,21921,21922,21936,21937,21938,21944,21945,21946,21947,21952,21953,21955,21963,21971,21972,21973,21974,21975,21976,21977,21978,21979,21980,21981,21985,21989,21992,21995,21998,22001,22002,22004,22005,22006,22010,22011,22012,22013,22014,22015,22018,22026,22029,22031,22034,22035,22036,22037,22038,22039,22040,22044,22049,22050,22052,22053,22054,22056,22057,22058,22059,22060,22061,22062,22063,22064,22065,22066,22067,22073,22074,22075,22076,22077,22078,22079,22080,22081,22082,22083,22084,22085,22086,22087,22088,22089,22090,22091,22093,22096,22097,22098,22099,22100,22101,22102,22103,22104,22111,22113,22123,22125,22126,22129,22134,22143,22146,22151,22153,22155,22156,22163,22170,22173,22177,22179,22188,22193,22198,22199,22205,22206,22214,22221,22223,22234,22237,22239,22241,22243,22245,22246,22247,22255,22259,22262,22263,22266,22268,22270,22271,22273,22275,22286,22288,22289,22295,22297,22298,22300,22301,22304,22305,22313,22315,22317,22322,22325,22331,22332,22333,22342,22343,22347,22349,22350,22358,22359,22371,22376,22378,22379,22383,22395,22401,22402,22403,22404,22405,22407,22417,22422,22425,22433,22434,22439,22447,22449,22456,22458,22462,22465,22469,22470,22471,22474,22477,22481,22484,22485,22487,22495,22501,22506,22508,22509,22515,22521,22522,22523,22524,22531,22533,22534,22537,22540,22547,22552,22553,22558,22563,22565,22567,22579,22580,22581,22583,22584,22588,22590,22591,22592,22593,22594,22596,22605,22606,22607,22612,22613,22620,22621,22626,22630,22636,22637,22638,22640,22642,22644,22651,22652,22653,22659,22660,22663,22678,22680,22688,22690,22693,22704,22708,22713,22714,22715,22716,22717,22718,22722,22728,22731,22732,22739,22740,22757,22759,22761,22763,22764,22765,22771,22774,22777,22778,22780,22786,22787,22788,22794,22799,22800,22806,22811,22815,22835,22837,22841,22860,22863,22868,22870,22877,22878,22890,22894,22896,22898,22900,22916,22917,22919,22926,22938,22941,22946,22948,22949,22954,22961,22962,22966,22968,22977,22979,22981,22982,22984,22985,22991,22994,22998,23006,23007,23008,23012,23021,23027,23028,23031,23032,23037,23038,23039,23046,23049,23054,23061,23069,23072,23073,23077,23078,23079,23080,23081,23082,23083,23089,23090,23094,23095,23098,23101,23103,23105,23109,23110,23113,23115,23121,23122,23130,23131,23135,23137,23139,23140,23152,23153,23154,23157,23161,23162,23165,23166,23167,23169,23170,23180,23192,23195,23202,23203,23204,23207,23208,23211,23212,23213,23214,23221,23222,23226,23229,23232,23233,23235,23236,23237,23239,23244,23246,23255,23259,23261,23270,23274,23281,23290,23300,23301,23304,23305,23310,23311,23316,23318,23320,23321,23327,23328,23340,23341,23345,23346,23353,23359,23365,23373,23376,23381,23386,23389,23391,23392,23401,23405,23407,23415,23420,23422,23423,23425,23427,23429,23433,23436,23437,23439,23442,23444,23445,23446,23455,23457,23458,23468,23469,23470,23481,23488,23491,23494,23496,23497,23501,23503,23506,23508,23510,23511,23522,23524,23525,23531,23532,23533,23534,23536,23538,23543,23544,23545,23555,23556,23557,23559,23560,23564,23571,23572,23582,23586,23590,23600,23611,23614,23616,23617,23620,23631,23640,23647,23649,23652,23653,23657,23658,23666,23667,23672,23675,23683,23687,23698,23699,23700,23702,23711,23716,23719,23721,23727,23728,23731,23732,23733,23739,23746,23758,23759,23769,23773,23778,23784,23796,23798,23799,23801,23805,23806,23807,23809,23810,23811,23812,23813,23824,23826,23827,23833,23834,23837,23840,23842,23843,23849,23854,23862,23863,23864,23867,23870,23873,23883,23886,23890,23893,23913,23915,23918,23924,23928,23933,23937,23941,23945,23949,23954,23960,23962,23963,23967,23971,23975,23977,23982,23983,23984,23985,23986,23995,24003,24004,24007,24012,24013,24016,24022,24027,24034,24036,24038,24040,24043,24044,24046,24051,24054,24060,24062,24063,24065,24069,24070,24079,24086,24087,24092,24097,24098,24104,24105,24109,24113,24116,24117,24118,24119,24120,24124,24132,24135,24147,24150,24152,24153,24156,24171,24172,24173,24178,24187,24189,24190,24194,24197,24199,24200,24202,24203,24205,24208,24222,24226,24240,24257,24260,24261,24264,24266,24268,24269,24274,24275,24277,24278,24279,24285,24286,24292,24294,24297,24298,24302,24304,24306,24310,24312,24313,24317,24319,24321,24323,24325,24344,24347,24350,24353,24354,24356,24368,24375,24383,24387,24394,24396,24397,24405,24406,24408,24412,24420,24426,24430,24436,24437,24438,24439,24442,24445,24447,24453,24455,24456,24460,24461,24462,24463,24464,24466,24477,24479,24482,24487,24492,24496,24497,24499,24500,24502,24503,24505,24507,24511,24514,24516,24517,24520,24524,24526,24527,24533,24536,24539,24540,24547,24549,24550,24553,24556,24558,24564,24565,24566,24570,24574,24575,24577,24586,24588,24594,24598,24605,24607,24611,24612,24613,24614,24616,24623,24624,24626,24631,24633,24635,24646,24648,24657,24661,24663,24671,24672,24681,24689,24690,24701,24704,24705,24706,24707,24708,24710,24711,24712,24714,24715,24716,24717,24718,24719,24720,24721,24722,24723,24724,24727,24732,24745,24750,24754,24756,24757,24766,24767,24775,24776,24778,24779,24780,24782,24783,24784,24788,24795,24798,24802,24804,24807,24808,24810,24811,24812,24819,24824,24826,24837,24842,24844,24851,24852,24853,24854,24855,24868,24871,24874,24877,24884,24893,24898,24905,24906,24915,24917,24919,24925,24930,24946,24947,24948,24952,24959,24962,24963,24969,24974,24975,24981,24982,24984,24985,24994,24995,24997,24998,25000,25002,25014,25019,25020,25023,25027,25030,25036,25037,25038,25039,25042,25046,25050,25053,25061,25067,25071,25072,25077,25082,25090,25091,25092,25095,25097,25106,25107,25112,25116,25117,25123,25128,25133,25136,25138,25145,25147,25150,25153,25165,25168,25169,25174,25176,25178,25186,25197,25200,25201,25202,25211,25219,25222,25223,25229,25230,25233,25238,25241,25256,25258,25267,25273,25276,25281,25293,25296,25299,25305,25310,25311,25314,25317,25319,25321,25324,25325,25326,25330,25344,25349,25359,25361,25370,25375,25378,25379,25380,25383,25386,25393,25395,25401,25406,25412,25413,25416,25427,25455,25458,25459,25471,25478,25487,25488,25493,25496,25501,25503,25507,25512,25513,25515,25527,25530,25534,25540,25552,25553,25554,25562,25568,25570,25573,25574,25577,25578,25600,25603,25604,25610,25618,25634,25641,25646,25650,25653,25662,25666,25673,25674,25688,25690,25704,25715,25726,25728,25733,25734,25736,25754,25755,25756,25770,25830,25836,25865,25869,25900,25906,25912,25937,25951,25954,25961,25969,25981,26023,26043,26045,26072,26075,26091,26147,26165,26168,26176,26179,26193,26194,26202,26203,26210,26226,26235,262

Viewing 5 replies - 1 through 5 (of 5 total)

  • shanebp
    Moderator

    @shanebp

    That query looks for users whose status is zero, iow. inactive or ‘dead’.
    Why do you have so many inactive users on your site?
    You could try deleting them or setting that field to 1.

    Re hosting – you are in a shared environment. Therefore resources are throttled. And queries can take a lot longer in that environment because of that throttling. WPengine can easily host very large BP sites, but not on the cheapest plan.

    So, you can clean-up your users and / or increase your hosting resources.


    r-a-y
    Keymaster

    @r-a-y

    The problem is with the function used to generate the active member count.

    On your site, you have a ton of inactive users, which is causing the SQL query to be very large. I’m guessing you installed BuddyPress on a really big site recently and your users haven’t logged in yet? I’m guessing your site has a ton of pending users that you haven’t deleted yet?

    BuddyPress can do better here, but in the meantime, one way to mitigate this SQL query is to generate a script that will backfill an activity timestamp for all your inactive users one way to mitigate this query is to delete all your pending users (the ones that haven’t completed registration) like Shane mentioned. This will stop WPEngine from killing the query.


    jaykdoe
    Participant

    @jaykdoe

    Thank you both for your responses. My website is quite popular, and like most popular online communities there are definitely quite a few accounts that have not been activated for one reason or another. I would have cleaned these up already but I don’t see an easy way to do that with BuddyPress, and honestly wouldn’t really expect that do break the site altogether. How exactly do I go about batch processing the deletion of inactive users? If BuddyPress doesn’t function well with a large number of inactive accounts, I would think there would be an efficient way to manage them in order to prevent the entire site slowing down to a crawl because of them. Simply deleting them manually would take an insane amount of time. I have to believe there’s a better way. Thanks again!


    jaykdoe
    Participant

    @jaykdoe

    Also, just to get some clarity, would inactive users be causing BuddyPress to slow down all functions on my website, such as loading a member profile, or the activity page? Because the entire site loads incredibly slow on every page, until I disable buddypress. Then it seems to function well.


    r-a-y
    Keymaster

    @r-a-y

    You can check the “Users > Pending Accounts” page in the WordPress admin dashboard. Then, you should be able to just delete them from there.

    It might be worthwhile for BuddyPress to have a built-in script that removes pending accounts that are older than a month or so.

    Also, just to get some clarity, would inactive users be causing BuddyPress to slow down all functions on my website, such as loading a member profile, or the activity page?

    Not really.

    If you indeed do have a popular website with many users that are concurrently logged in and performing operations, then this will slow down your site. You will need to utilize object caching plugins and the like (I believe WPEngine already has something built in for this). You might also need to consider upgrading your WPEngine plan.

    We’re also thinking of doing some performance enhancements in the next version of BuddyPress that will decrease the amount of code that needs to run on non-BuddyPress pages.

Viewing 5 replies - 1 through 5 (of 5 total)
  • You must be logged in to reply to this topic.
Skip to toolbar