[Xerte-dev] Re: example_zipper issue

[Pat L (pgogy)]

This has been there from day one. We had Nottingham users before it was
open source who had this problem.

David has fixed size, but timeout might also happen as well.

Why are people exporting? Is it to edit elsewhere or to use? If it is edit
then maybe they don't need all of it? Leave media relative export? If it is
to use then maybe making media into links makes sense. Smaller zip?
>
> On 2 May 2014 11:59, "Ron Mitchell" <ronm at mitchellmedia.co.uk> wrote:
>>
>> Hi all
>> My inclinations is to scream aaaarrrrrgh! Having had what seems like
>> coincidental support requests but is clearly all part of the same issue
so
>> I've lost track of how long this issue has existed and how widespread it
>> might be. To explain...
>>
>> Late yesterday I received a request from one of the HEA projects
regarding
>> two of their students being unable to export their projects. After
>> investigation it turns out both students have been uploading quite a few
>> videos as part of their LO's in this case .flv's and so sure enough the
>> projects weren't exporting and creatining more and more example_zipper
files
>> after each attempt.
>>
>> One of these particular projects showed 158MB of media via media & quota
and
>> the other 242MB
>> One had 47 zipper files the other 21 and across all the user folders 200+
>> zipper files!
>>
>> I'm a bit concerned how widespread this could be with people unaware of
the
>> issue. I know it's not best practice to upload loads of video direct but
>> lets be realistic users are going to do this and sometimes with good
reason.
>>
>> John - This server was also set to 128MB memory limit by defauls so I
tried
>> your export.php but that was generating an empty zip download.
>>
>> I've increased the memory limit to 300MB and with the current 2.1 code
and
>> original export.php the above two projects export ok without leaving
>> example_zipper files behind.
>>
>> However I also noticed looking at the folder via ftp as you suggested
John,
>> that if the user cancels the export, even with the memory limit
increased,
>> the example zipper doesn't then get removed. I don't know if this would
be
>> different with develop or if your export.php would only work with develop
>> John?
>>
>> But I am wondering if this is a bit of a ticking timebomb for lots of
>> installations! If anyone has SSH access to their installation(s) try
find .
>> -type f -name 'example_zipper*'
>>
>> Perhaps the temp file name should have been example_zippo! ;-)
>>
>> We obviously need to crack this one asap...
>>
>> Ron
>>
>> -----Original Message-----
>> From: xerte-dev-bounces at lists.nottingham.ac.uk
>> [mailto:xerte-dev-bounces at lists.nottingham.ac.uk] On Behalf Of Smith,
John
>> Sent: 01 May 2014 14:51
>> To: For Xerte technical developers
>> Subject: [Xerte-dev] Re: example_zipper issue revisited
>>
>> Hi Tom,
>>
>> Yes, definitely, no worries. I'm about to investigate a windows install
with
>> both the original code and this new code to see whether I can get it to
run.
>> Will update you.
>>
>> Regards,
>>
>> John Smith | Learning Technologist
>> Room A251, Govan Mbeki Building | School of Health & Life Sciences |
Glasgow
>> Caledonian University Cowcaddens Road | Glasgow | G4 0BA
>>
>> Please address ALL support requests to hlsblt at gcu.ac.uk where one of the
>> School Learning Technologists will pick up the job. This will ensure that
>> all jobs are completed as promptly as possible.
>> ________________________________________
>> From: xerte-dev-bounces at lists.nottingham.ac.uk
>> [xerte-dev-bounces at lists.nottingham.ac.uk] On Behalf Of Tom Reijnders
>> [reijnders at tor.nl]
>> Sent: 01 May 2014 14:46
>> To: For Xerte technical developers
>> Subject: [Xerte-dev] Re: example_zipper issue revisited
>>
>> I will look into this, but I don't have much time at the moment. Perhaps
>> topnight. Otherwise, it'll have to be next week.
>>
>> Tom
>>
>> Smith, John schreef op 1-5-2014 15:38:
>> > Well. Preliminary testing looks good with my new code but i haven't
tested
>> on Windows yet - would be interested in getting Tom's take on all of
this...
>> >
>> > I just added loads of videos via ftp (upload limit set to 7M on this
>> > server) and checked the memory limit and it is
>> >
>> > memory_limit    90M     90M
>> >
>> > and with each video being used on a page then i've just successfully
>> downloaded a 316Mb archive zip without any problems (it took several
minutes
>> too so the script didn't timeout) and was able to play the videos
contained
>> within... and this is just a cheap Dreamhost shared server, nothing
fancy...
>> >
>> > On to Windoze...
>> >
>> > Regards,
>> >
>> > John Smith | Learning Technologist
>> > Room A251, Govan Mbeki Building | School of Health & Life Sciences |
>> > Glasgow Caledonian University Cowcaddens Road | Glasgow | G4 0BA
>> >
>> > Please address ALL support requests to hlsblt at gcu.ac.uk where one of
the
>> School Learning Technologists will pick up the job. This will ensure that
>> all jobs are completed as promptly as possible.
>> > ________________________________________
>> > From: xerte-dev-bounces at lists.nottingham.ac.uk
>> > [xerte-dev-bounces at lists.nottingham.ac.uk] On Behalf Of Ron Mitchell
>> > [ronm at mitchellmedia.co.uk]
>> > Sent: 01 May 2014 10:58
>> > To: 'For Xerte technical developers'
>> > Subject: [Xerte-dev] Re: example_zipper issue revisited
>> >
>> > Hi John
>> > the problem is when I originally raised this issue on 8th April it
>> > coincided with discovering loads of these un-removed example_zipper
>> > files on the sandpit install. At that time I also mentioned the
>> > separate session issue as a reason I hadn't updated to a newer
>> > revision and I think you mentioned that you thought Tom had fixed this
>> > zipper issue in a later revision. So once we'd identified the cause of
>> > the session issue (or at least Julian's attempt at fixing his FF
issue) I
>> updated the sandpit code as well as other servers.
>> > Now I've been using the 2.1 branch for this rather than develop so I'm
>> > not sure if Tom's fix is in develop but not 2.1 but I think the code
>> > is the same at the moment.
>> >
>> > In short although I've been using a 62MB file from a project a user
>> > was unable to export, previously there were loads of these zipper
>> > files in loads of different user/project folders and there are
>> > probably quite a few again now prior to my changing the memory limit.
>> >
>> > I could try resetting the memory_limit back on my own server and test
>> > again but my previous tests with the 62MB file certainly weren't
>> > crashing the server. Let me know if you think its worth testing with
>> > larger files and the default 128MB limit or if your and David's later
>> > responses have ruled that out as being worthwhile?
>> >
>> > I know Tom said last night that he was going to revisit his fix for
>> > this so would be good to hear back from Tom once he's done that.
>> >
>> > Cheers
>> > Ron
>> >
>> > -----Original Message-----
>> > From: xerte-dev-bounces at lists.nottingham.ac.uk
>> > [mailto:xerte-dev-bounces at lists.nottingham.ac.uk] On Behalf Of Smith,
>> > John
>> > Sent: 01 May 2014 09:57
>> > To: For Xerte technical developers
>> > Subject: [Xerte-dev] Re: example_zipper issue revisited
>> >
>> > Hi Ron,
>> >
>> > Have been thinking about your issue and just wondering whether you
>> > might have inadvertently chosen the worst file size possible for the
>> > video with 128Mb memory available to PHP. Let me explain (and this is
>> > entirely hypothetical)...
>> >
>> > The zipper creates a blank object (<0.1Mb memory used) The library
>> > loads all the other files individually and zips them into the object
>> > (0Mb -> compressed size of all js/html is perhaps around 1-2Mb or
>> > under) The library then loads the mp4 (now it has the object = 2Mb and
>> > the video = 62Mb in
>> > memory) It adds it to the zip object but it's also still in memory
>> > (mp4 doesn't compress at all so we have 2 + 62 + 62 =126Mb which is
>> > less than
>> > 128Mb) It then closes the connection to the mp4 and destroys the
>> > object (memory now = 64Mb for the zip)
>> >
>> > I'm wondering whether a 64 Mb or slightly bigger will cause you to be
>> > able to crash that server too. Is that something you could try?
>> >
>> > Regards,
>> >
>> > John Smith | Learning Technologist
>> > Room A251, Govan Mbeki Building | School of Health & Life Sciences |
>> > Glasgow Caledonian University Cowcaddens Road | Glasgow | G4 0BA
>> >
>> > Please address ALL support requests to hlsblt at gcu.ac.uk where one of
>> > the School Learning Technologists will pick up the job. This will
>> > ensure that all jobs are completed as promptly as possible.
>> > ________________________________________
>> > From: xerte-dev-bounces at lists.nottingham.ac.uk
>> > [xerte-dev-bounces at lists.nottingham.ac.uk] On Behalf Of Ron Mitchell
>> > [ronm at mitchellmedia.co.uk]
>> > Sent: 01 May 2014 09:09
>> > To: 'For Xerte technical developers'
>> > Subject: [Xerte-dev] Re: example_zipper issue revisited
>> >
>> > Hi John
>> > I was just going to test your export code on a server where I don't
>> > have access to change php settings so the memory limit is fixed at
>> > 128MB. I know what David means that ideally we want cross platform
>> > code but as you say there are inevitably different
>> > solutions/restrictions for some e.g. no control over memory limit. In
>> > fact the upload limit on this particular shared server is set at the
>> > default 2MB so I had to upload the test 62MB mp4 via ftp. However I'm
>> > now confused further because this is the same XOT code as the other
>> > servers I was using for testing so I expected the export to fail when
>> > testing before replacing the export code with yours and yet it didn't
>> > fail and I was able to download the zip which didn't contain any
>> example_zipper files.
>> >
>> > I'm glad you're able to reproduce the issue but not sure how a server
>> > with 128MB memory limit is able to export ok without a change of code
>> > or memory limit :-(
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: xerte-dev-bounces at lists.nottingham.ac.uk
>> > [mailto:xerte-dev-bounces at lists.nottingham.ac.uk] On Behalf Of Smith,
>> > John
>> > Sent: 01 May 2014 08:38
>> > To: xerte-dev at lists.nottingham.ac.uk
>> > Subject: [Xerte-dev] Re: example_zipper issue revisited
>> >
>> > All i know Dave is that, with the commented code uncommented, my
>> > system couldn't download a 56Mb an 79Mb zip but with this code 79Mb
>> > came down no problem.
>> >
>> > We may have to use what we have access to on different systems. There
>> > may be a similar way to pipe the result into windows or we fall back
>> > to the old code and crank the memory requirements up yo 300Mb which
>> > not everyone can most likely do anyway...
>> >
>> > Regards
>> >
>> > John Smith
>> > Learning Technologist
>> > School of Health and Life Sciences
>> >
>> > Sent from Samsung Galaxy SII
>> >
>> >
>> >
>> > David Goodwin <david at palepurple.co.uk> wrote:
>> >
>> >
>> >
>> > <snip>
>> > Uses the unix command zip and pipes the stream back into PHP in 8192
>> > chunks
>> >
>> > <snip>
>> > Also, might not work on Windows...
>> >
>> > So therefore it's not worth using?
>> >
>> >
>> > Technically there is no benefit to doing the fread(....) over a
>> > fpassthru(...) fpassthru does something like 8192 chunks internally.
>> >
>> >
>> > David.
>> >
>> > --
>> > --
>> > Pale Purple Ltd
>> > http://www.palepurple.co.uk @PalePurpleLtd
>> > 0845 0046746 / 07792 380669
>> > "Bespoke web and mobile application development"
>> >
>> >
>> > This message and any attachment are intended solely for the addressee
>> > and may contain confidential information. If you have received this
>> message in
>> > error, please send it back to me, and immediately delete it.   Please
do
>> not
>> > use, copy or disclose the information contained in this message or in
>> > any attachment.  Any views or opinions expressed by the author of this
>> > email do not necessarily reflect the views of the University of
>> Nottingham.
>> >
>> > This message has been checked for viruses but the contents of an
>> > attachment may still contain software viruses which could damage your
>> > computer system, you are advised to perform your own checks. Email
>> > communications with the University of Nottingham may be monitored as
>> permitted by UK legislation.
>> >
>> >
>> > Glasgow Caledonian University is a registered Scottish charity, number
>> > SC021474
>> >
>> > _______________________________________________
>> > Xerte-dev mailing list
>> > Xerte-dev at lists.nottingham.ac.uk
>> > http://lists.nottingham.ac.uk/mailman/listinfo/xerte-dev
>> > This message and any attachment are intended solely for the addressee
>> > and may contain confidential information. If you have received this
>> message in
>> > error, please send it back to me, and immediately delete it.   Please
do
>> not
>> > use, copy or disclose the information contained in this message or in
>> > any attachment.  Any views or opinions expressed by the author of this
>> > email do not necessarily reflect the views of the University of
>> Nottingham.
>> >
>> > This message has been checked for viruses but the contents of an
>> > attachment may still contain software viruses which could damage your
>> > computer system, you are advised to perform your own checks. Email
>> > communications with the University of Nottingham may be monitored as
>> permitted by UK legislation.
>> >
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > Xerte-dev mailing list
>> > Xerte-dev at lists.nottingham.ac.uk
>> > http://lists.nottingham.ac.uk/mailman/listinfo/xerte-dev
>> > This message and any attachment are intended solely for the addressee
>> > and may contain confidential information. If you have received this
>> message in
>> > error, please send it back to me, and immediately delete it.   Please
do
>> not
>> > use, copy or disclose the information contained in this message or in
>> > any attachment.  Any views or opinions expressed by the author of this
>> > email do not necessarily reflect the views of the University of
>> Nottingham.
>> >
>> > This message has been checked for viruses but the contents of an
>> > attachment may still contain software viruses which could damage your
>> > computer system, you are advised to perform your own checks. Email
>> > communications with the University of Nottingham may be monitored as
>> permitted by UK legislation.
>> >
>> >
>> >
>> >
>> >
>> > Glasgow Caledonian University is a registered Scottish charity, number
>> > SC021474
>> >
>> > _______________________________________________
>> > Xerte-dev mailing list
>> > Xerte-dev at lists.nottingham.ac.uk
>> > http://lists.nottingham.ac.uk/mailman/listinfo/xerte-dev
>> > This message and any attachment are intended solely for the addressee
>> > and may contain confidential information. If you have received this
>> message in
>> > error, please send it back to me, and immediately delete it.   Please
do
>> not
>> > use, copy or disclose the information contained in this message or in
>> > any attachment.  Any views or opinions expressed by the author of this
>> > email do not necessarily reflect the views of the University of
>> Nottingham.
>> >
>> > This message has been checked for viruses but the contents of an
>> > attachment may still contain software viruses which could damage your
>> > computer system, you are advised to perform your own checks. Email
>> > communications with the University of Nottingham may be monitored as
>> permitted by UK legislation.
>> >
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > Xerte-dev mailing list
>> > Xerte-dev at lists.nottingham.ac.uk
>> > http://lists.nottingham.ac.uk/mailman/listinfo/xerte-dev
>> > This message and any attachment are intended solely for the addressee
and
>> may contain confidential information. If you have received this message
in
>> error, please send it back to me, and immediately delete it.   Please do
not
>> use, copy or disclose the information contained in this message or in any
>> attachment.  Any views or opinions expressed by the author of this email
do
>> not necessarily reflect the views of the University of Nottingham.
>> >
>> > This message has been checked for viruses but the contents of an
>> > attachment may still contain software viruses which could damage your
>> computer system, you are advised to perform your own checks. Email
>> communications with the University of Nottingham may be monitored as
>> permitted by UK legislation.
>> >
>> >
>> >
>> >
>> > Glasgow Caledonian University is a registered Scottish charity, number
>> > SC021474
>> >
>> > _______________________________________________
>> > Xerte-dev mailing list
>> > Xerte-dev at lists.nottingham.ac.uk
>> > http://lists.nottingham.ac.uk/mailman/listinfo/xerte-dev
>> > This message and any attachment are intended solely for the addressee
and
>> may contain confidential information. If you have received this message
in
>> error, please send it back to me, and immediately delete it.   Please do
not
>> use, copy or disclose the information contained in this message or in any
>> attachment.  Any views or opinions expressed by the author of this email
do
>> not necessarily reflect the views of the University of Nottingham.
>> >
>> > This message has been checked for viruses but the contents of an
>> > attachment may still contain software viruses which could damage your
>> computer system, you are advised to perform your own checks. Email
>> communications with the University of Nottingham may be monitored as
>> permitted by UK legislation.
>> >
>> >
>> >
>> >
>>
>> --
>> --
>>
>> Tom Reijnders
>> TOR Informatica
>> Chopinlaan 27
>> 5242HM Rosmalen
>> Tel: 073 5226191
>> Fax: 073 5226196
>>
>>
>> _______________________________________________
>> Xerte-dev mailing list
>> Xerte-dev at lists.nottingham.ac.uk
>> http://lists.nottingham.ac.uk/mailman/listinfo/xerte-dev
>> This message and any attachment are intended solely for the addressee and
>> may contain confidential information. If you have received this message
in
>> error, please send it back to me, and immediately delete it.   Please do
not
>> use, copy or disclose the information contained in this message or in any
>> attachment.  Any views or opinions expressed by the author of this email
do
>> not necessarily reflect the views of the University of Nottingham.
>>
>> This message has been checked for viruses but the contents of an
attachment
>> may still contain software viruses which could damage your computer
system,
>> you are advised to perform your own checks. Email communications with the
>> University of Nottingham may be monitored as permitted by UK legislation.
>>
>>
>>
>>
>>
>> Glasgow Caledonian University is a registered Scottish charity, number
>> SC021474
>>
>> _______________________________________________
>> Xerte-dev mailing list
>> Xerte-dev at lists.nottingham.ac.uk
>> http://lists.nottingham.ac.uk/mailman/listinfo/xerte-dev
>> This message and any attachment are intended solely for the addressee and
>> may contain confidential information. If you have received this message
in
>> error, please send it back to me, and immediately delete it.   Please do
not
>> use, copy or disclose the information contained in this message or in any
>> attachment.  Any views or opinions expressed by the author of this email
do
>> not necessarily reflect the views of the University of Nottingham.
>>
>> This message has been checked for viruses but the contents of an
attachment
>> may still contain software viruses which could damage your computer
system,
>> you are advised to perform your own checks. Email communications with the
>> University of Nottingham may be monitored as permitted by UK legislation.
>>
>>
>>
>>
>>
>> _______________________________________________
>> Xerte-dev mailing list
>> Xerte-dev at lists.nottingham.ac.uk
>> http://lists.nottingham.ac.uk/mailman/listinfo/xerte-dev
>> This message and any attachment are intended solely for the addressee
and may contain confidential information. If you have received this message
in error, please send it back to me, and immediately delete it.   Please do
not use, copy or disclose the information contained in this message or in
any attachment.  Any views or opinions expressed by the author of this
email do not necessarily reflect the views of the University of Nottingham.
>>
>> This message has been checked for viruses but the contents of an
attachment
>> may still contain software viruses which could damage your computer
system, you are advised to perform your own checks. Email communications
with the University of Nottingham may be monitored as permitted by UK
legislation.
>>
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nottingham.ac.uk/pipermail/xerte-dev/attachments/20140502/1c2b24f7/attachment-0001.html>