Issues with media preview and sessions since 1.8-RC3 upgrade

Hey guys,

Ever since the 1.8-RC3 upgrade on Docker we’ve been having issues loading preview thumbnails of media. They usually, but not always, will result in broken images. Sometimes this also results in the session ending.

Attempting to load a preview of an image directly via URL results in something like this:

‰PNG  IHDRF8@Ž5	pHYsÄÄ•+eÆIDATh½›{t×}ß?ß»ƒár±A)˜¦€} ES²BêeJq“Ô­åÔnjGNRç$Žœ¦±-¿ää4vÒœ6ǍÝX©ERì$¶eÇ­âÆŽ#+Šä©Ž¥Ø¥È*Eâ±»‚ —Ëår±œ¹¿þ1³H‰Ô#içà`çÎ̝{ó{ÿ¾#3ã¥H’33ßý¿zì%oXEcÅR8Y­tÆŠ¥¬ad\†L&Óéœ=›sRä}2…AgºV}Ùù^u×Z.r@ÔôD±÷n²2½’½pÿyÃŽ,·Û¡äœmoq(ä€,Ðq×2o93s@´·_Ú>6{8ׄ§'&þ)–@ñÒá|OØÓŽã8;]«5_ËÿhÆŒKAì½Ã,+ÉIjyï³rʛы„:$ò}` çœ3oä­é™Zë³–ÒðÈV S™Y8ÿÜî+®¤~òDÐÓÓå×õùÇßÁ¹t!Uz9+–rqG™L&0³À9çr¹\ãÔ©Sý©˜ˆ<Ð"¡¬ae1êˆÀŒœ .©nà0‘üt­Z5ë(ÛäT™ª&ªY)tוCDëÖ­o6u7]«E¯dÞ‹JL×Ƭe/•ƒ8Ž³@e‘BKm†¤-fD‚&¢‰‘¶a¡äZ˜mÃv3,Zšž©½*/·Â.à•Ê#…<Ð!1°¡¤~3kãÉèò† aä!ê5³~D†“Ô4³J½tÑ[1"¤¹„1¶¨¨Q[Òa`!µCs†mÃø6b”sÎ=ìãø$}wݺuaN5OåÌxØB>×ûè÷> @qx$1IéÜuIÞ’¸«³gÏîæcíÏf2hzªsQÆ”G !‰·ŠRɨeô+95l7èˆ`£™ " ü¨¬OhcªNý’êf6„ιï}¿œšfÖuR7]–k`d‘¦»c‰ÄFõ#݇ٻÃ0üìÙèlhÞn²Õó_/‡N.ôÞ_)ÉeVš03W=<Û9Ÿ«éŽ;îà¿}æ6×ß·Þïû‡Ç_ì®Ë#…À D†mZ4³-$º¥aG0®®¾öºëŽ]síµ[———·AàæžyÆ=õÔ÷û¦¦¦rB™ ¡¤ÈÌÜàæͼîu¯s===‘$ŸÉ¸v§s–l6ÛYî,Ï>ö½}Kˆy`)aD²8Žq-ðIàÍ.ã¾`Þ f¶SÒ½‰„ª¶„ÑÌd2µ©Z5‚wM=(===8ŽsHí©jůÞû½Rêr–Ä}fæ$af ifyDáÇo¼ñý¿xóÍ»×ærî’K.át³É¦ÌŒèlľ}ß«äƒÊŸ8q"XýVþâ/¿ÎevíÀ¼!'Îv:d‚€¯ÿÅ×øÕ[oM™QA´1¶!¦0v"¾˜ªÄ·Ìì`VÒ3ûià»À¾pÍšN§³¼ã ÷_j™Ù¢ŽÑtÎ5̬É)‰±ê@Þ9ÅqÜéééñ‡¦§8gácÅRè½ïHê&{Þ›•Ò€¬€p?ÿïÙö>þë»çã˜Æ©SnÞ¼2GìÙ³§ÿ–ÈÿÇOüÆÊøÞe®g×å—“¾\&ÑàƒaR¯×9zô(Hƒ`¹ô¥’ƒ,²oaü3Iifïîþeêÿk*9{Ïv:!FSÒÃÝà0}üÜyš³t^vÝ/—ËdØ9¾ý‰/•™Þ,4ïC9e1ú»ûMu½ñ½ýÝºqÓ¦kΞ=KÇ<¿ð<¯ßöúsžÚétˆã˜½×^Kãd€¯Üó?Ù½gçS¥RáñÇösòäI>ý;¿Ã*oæ€ÒIÞUÁ”ô!3»EÒ2³]À°¤G)I-o~aF	Ñš"IE¶’x¿¦sn6±‹òNê¬]»váôéÓáôLíœÔaEb\&CÇidk>U#,Ú´…Booï5===„aÈñãÇÙ4°‰\.‡$þîÛßæñýûyßa>ÀÈHï?ù${®ºŠÝ{öªåÊÃ%ñÜü<ǏaÜHD OåN¥L¸ßÌþ-ðe3ûâ+5à @Á¼µ€9çܽSµêêLýüÄk¶ûcçøva¸±R¹³h²Zi+F·PÈ¥Ìͬ#i@(kf?AOÏÎL&³²ÁS§Nñeÿ8Üô®›ÜÌ•?x%¿ôË¿Œ$²Ù5üû÷ý ’V˜ñ÷>ʵ×]÷ƒœZ½ðÕž²mfÃÀàJ`¸ñ Æۏ	}ð&ëÇðÞ¿»42Òæ%3o¢_R˼Ճ ˜÷Þ÷9çæ'«Hâ¦îúÜx©c¥æ}€¢…1”^—5l/ða¤Gƒ hj6Ù°a>ò‡à«þUžž8„sÉžŽ;Æâó‹ìºürö^ýÊ.ã8æî;ïZaÌ%—\ÂäÄáš5¼9’0’ø	Ä£Wc4$ýµ™½É°a’ˆ¸&é©÷èƒûù»ïæꫯãá‡ÿ6¨ž9§Ôpý5×0ÿÜ‚}¼µWUffgøƒ;ïJ͐Ä7ï½—jµ²r݉'ؼeÕJ•õë×sÛg?KöðÈwáÎ;îèNïIò§<Ə"¾Œñ3û¦¤¿J½æ8°Í`áöÛ~¯efý=ô-‡Ñ, Ô$aG}áùÅ©Ô¦33¿}tÔ™Y{´PÌÑT­ÚÌ̏•J^N”)}$¶'TRqÍ8!é`ãäÉè7>þñÒjîoØ°OæwÙ{ýõ+êbfü¯|…r¹ÌÿètEï=wýÁœ=e­Œe³YÖ®]KOO@¸&äúeéZ\\$è	ó­V‹Ù™™Á\oïÜúõë‡zz‚·?¿ðü¾7íÝû®§¸õÙgŸ=–2.ÂlÀÒMaØ}B¿[9<;[ÉÇqÏÜ3slÚ´	Œ•ñþþ~¾qlŽ9€ÏÝ}7³³³C¿ýÉOlùÒ¿¸÷Ý?÷s|úw>ÅÝwÞ¹…—¦Æ"a¼³xéðι¹Éjåàª2F{´PÜ2Z(v€¥o?òwèåB!Äp„óæFÓ¤>°¬Ðãf¼ë|¦¼a×.îúÜ288xŽ§¹ï›ßäÖ„8ŽñæÏ9÷ú×oåÿèó„a¸2îœÃe{o¸Ñññ•ñk¯»çÜÊñ;oº‰Oþöo¯ÿ«·½ 3cyyÅnv€â˜Ð‰±ÞffO`ôeöFà‡bgK##KÅááCcÅÒ]ÓµZ˜ß^u£…bÎ9QrÎeœkyï·u Ã1n6ì’…~üÆeù/Ÿþk×®]YhÇüÞgnãî;ï\óþÜÌ}Çe—±}ÇŽsÆä –Žço|ãʽõ'¸ý³·óяÝJoo/¹\Ž(zÁ†®_¿3cßC`cÀ8ç¹?|ΓS ¤?ðÞ_W.¦¦kµ#‡¦§J³yŠ‹ÐBZòtξãÍoús9=1]«Í”G 香e§gj  ô¸L›À¹N§ã`ŒšÙm$É_v÷;áš0¸åƒ|ÑÓƒ àæ_zï‹Æë7“ååeÎvÎâ}LÜ-šý©Ë?¾tœZµF¹\&Žc¶nÝ @·)Çñ9svzï_ÝËáÙÙ‹1`Kê9Þû+pñ/¼÷7—F ·‹…ûþæ[ßZ+–:åB!?]«5ƒÈûÀâ84lÀá"Ãæ€o?¯ˆš%]mPϧÕÒðÕ{þœÿþÅ/½èšµk×òTZ3qC¯ebß÷¾Ç·ÿ>ßúkÖd$½HTæ…?€èìÙ—cJ—ò«~;IÍL&s4öþ!'¨Vª£…bslûxûe÷ÞëÆh¡˜sf–•s¤5ÔÐ øðħ¦=ÕsmÆòòò #º‹^QXýó)~¡ÝB6›åè‘£äz{Ï1Ô™4É\m¤½Á˜_hî—¡œ™FQô[˜}È{ŸŸž©-þÐÓaûG‹¥~€é™Z[£Åbˆ‘õÞo”SǼe-˜Ýl[a·sü›w¾#}‹ÐµdtÎvh>MO®?ôÐCÄÑ‹û[’Øuùå+êpªÑàðáÊEÌúáóÇ_üŸþԧؼy3Û.½óFG<ñø´—Û¯”1I’x0(éˆ%uà60 ¬Ã¬?Š¢¾ÑB±5U«vMM¶2?°q cæ½™%=¡¤¢µX‡T¬ClD:”1’a^•’_uõUì¸la¹Þ^'ed³køñeod×å—S­V`óæAF E$ñì³ÏÒÛÛËîÝ»Y^^æà§yðaعó ärk™©Õhžº !"1´5çÜŸHz°'¿†4	Ì!šNîÀ3™Læ¾é™Ú“›6l<+ÑsÇí·ëý·ÜbÁÏßüžÎçïº;””ÅKr¤&àÒ~P4‡îaíN“±WEgδ‘ÄÂsÏqúôi®ºújþâ«ÿ‹S§,/wø³?ýSÞñSïdíÚµ|õž{øáù&'&¨=žåêk¯á²;Yxî9Â5!›6mâÙ¹gW?¢MÒ÷îdÚàIà$õ=Ûy«yû(Ié'½ùêìl¥­w‚ ØÇqaèLÏÔÆKå¬7ËMU+Kã¥r(3c¼<šM¢T—%is8©ÙVð|Úðz=bc'¬”=ÿP×NäÓuÍS@CR?°¶z~‘´Ì"jØJJÐ!±3 Iǐ~—&Òœ¬VZ—¹8öù‰ÊtC’KzÌ€PnMØS_^îäIfÖuQ-Ä‚PdØN “FÊÝ´â‚$é.97­¤ðµÙÌ"Œ!ƒf£\œÉs$%…ÅtÃß–´Ã9w§™e{{{ïk6›Wȹ_1³a0{¹Læç1»Â›ßôIÁbê\”ÀLêIv]ZOONú·ßxcs¬XÊšYÛÜô³?ÓqÎùååNI—1òÞ³`}-P8eð,p?°_èiâu12³Ý˜uÌlÂÇþǜ܌¤9%¶`>eÞ‘ômΓ´f‘Äûï"žDú&⌤¿ñÞÿŒ™Ñjµò’ú2Î=(xéo3Aæ#À[ͬ-§Ï ž ‚àÏ$ùÞuùoMÕªaJÃ#%€éZun´PøZùvvŒŽ½Ð%(9—3óF˜¾Å¦a»…†H:‹åÄ+Y!íû¥öæœVÌKˆÍþŒsïñÞBҝÞûÏ9I÷˜ÙNoø6I¤kê 1'Óœ§.§>Œ7™ÙMÀ—ßwrsf6Ø›q™±÷… “ùFÇ»¼ùš”´di·aºV;'韪VæÆŠ¥¾Éj¥°½<:%¸9çœÌû–'Ñé–a9¡	ŒÃ–Riê`v c1‡Qaº‹ˆÍnïý¥¥È[ÀÀŒ¤¯[ÃOOb–tXh©žöÃ)eÊב¾,tÀÌæ€:ÆÃÞû%'í‹¢è&o~OØ~ åÍo‘ôp&“™-SµjÇÌüî7^é$¹ŒsÍÑB1ì°2$&†ã'NÄeû7ôJ:#iaêb趝Ö+ÁµÄÀ&Äq` ‰iK2;@wÞóèGͬ(é~9½_¨NÒÊh§qÒàËrzNèIG ê`m¡Y¡çSizTÒçëG“ØŠ¦¤C†]b˜ÌìÀSµê·7õox^訤pªV=¹iÿ±¿_•Ù™Æº\oÿñ'Ú‹KKlÚ¸Á-8ᏟX²•Òf*5mï}VRG¨Ã=Å9ÓÒF°›a1:À€™uaf9X1Êçeæ]fVÀØ éïäô¨yہÙ[ý’SƒÞNj$`$v¶ i†áHúEe“’(‹Àh¾2S›- ¼#íGyI‹Åá‘áñR™(Ž[J rLV+Kã¥rßdµÒtr¾=ºmÊôg¼Tvq;¤N‚¤d^Ig°%ô”a£À0rHŽäþV ïJÅÅ›ÙJ7s£Bàƒfön<aÌì;À¤§Kâ&³-ˆ$¦2¤c’Ž¤]Ê…4Jr:`Þú$ê‚N¹Pì_¿áÀ¾'ö¯ìMI¯z)Š¢é™ÚÂh¡ØÇý¤öx©ìMOEÛË£‰§>¯æ@¹PÌ¥hÊ<æ[ m†5á³ëË¥ñÌÄ™´a¶É°| Û0Ë¥ê×gfƒ$v(aÌ¥viÁ°&†_µÐy Žt\‰QN€‹¢af‹[%ía†2Ap0Ž£>¡hªV­Ky3¦gjU½éÕ{Û6]«‘äFÅþÉjeiûè˜s‚§''W®Qü1^*‡ÓµjK›ù¦ä†æAaŠSY²ÄcõaOÕ keöV¯Þøh±– ¥è¾aÿæ%‘á/ÒײæSµjÇÉyç\‡¤Sʹ:ÐF4Ó`p	qh‚ZB ¡%Ú–¢½ó	s4G¢–M¡cˆÉGÎ{_ 5I´ðÔ$ùÁMÕªu3ú¦ªÕtë´òB«÷µúšéZ-š™­s~õ̯+ ®·7Ïéf3””G4½÷¡¤P(0ð™ŒkÆQ•ÊA÷‹·íÿBŒ¹Š¡aFàœË™Y;­ø…©¸vcˆ~DhI€·(Ô´²žEŠÒH¹›xæÒVj”BÛû‘"°¦™õ	"ç2Moä×å£'ž|òÿÉç‚«éBîúbÜsÏ=ѯì×Z8²A&hGQäÒ<)È™fÅŒ01°ŠHŒl@àÉ“Ô^efɇe’ºLI?å±–P^ε×õ­kŸ:Ù¦SìîkXó«¦Wý!×KQy¤f2™Ž÷>|sÞ›o¦õcgfnó–KêÏÍ?;H"Qa‹qçÒ.f@R¤ê#é372Aà~²ZyE_¤ýSÐ+v×›`õñT­Úž¨L3U«63A¦)§VefÆ{ï;H$¾ó÷xƒ†œ:’Ÿf¸Q6›­§ö¥õÖŸ¸q)“É0U­t^	S^ní¯õøEÈðÕ.kµÛsóE\äŽÑ1ç}ì,ÅìNV+í±R9Ä,뜚QaBN’Ÿ¬V:£…b8=S‹^ͳ^íù—;^ýûÿqúEÛµ­'QIEND®B`‚

I suspect that this may relate to the large size of our images. Are the previews generated on page load? It seems like there is no thumbnail cache, which may simply be taxing the system too hard to generate that many previews at once.

Thoughts?

This is interesting - we did remove the thumbnail cache in 1.8 - in favour of Etag based caching, so that each browser will only request a thumbnail once a week. This was done to allow us to serve different sizes of the same image without building a massive file cache.

It will generate thumbnails for a page at a time of any results - once that data has loaded (not exactly on page load, on table load).

I think I am happy to re-introduce the physical disk cache for these things - but I would be interested in seeing if there are any logs recorded for those thumbnail requests? (you can filter on the page download in logs).

Now that I know what you’re doing I see part of the problem… We run Xibo behind a reverse proxy for the purpose of running on SSL. At this time we’re using a privately signed certificate for the SSL, and ETags and last-modified headers do not work on untrusted certificates.

That said, based on what I’m seeing I don’t know that a full page of thumbnails could be generated at once to be cached in the first place.

They are generated by individual requests, which will be concurrent up to the number of requests the browser is allowed to make concurrently.

I do take your overall point and I think I am keen to put some short-term file cache back in place.