Hello.
We have two nginx serving static objects from a shared NFS partition,
and we are using open_file_cache for performance purposes:
With nginx 0.6.x, when the developers uploads files, during a few
seconds (less than 30s) we see this errors in logs:
[alert] 2721#0: *75303 sendfile() failed (116: Stale NFS file handle)
while sending response to client, client: XXX ....
I understand that is a normal behaviour because we are using a open_file_cache.
But, when we upgrade to nginx 0.7.59 the same errors appears
indefinitely (and of course we are sending corrupted objects or
blank), it seems that, open_file_cache_valid is been ignored in new
version.
Any idea, tip or recomendation ?
In other hand, is recomended the use of open_file_cache, until now, we
are very happy with it.
Thanks in advance.
BR.
On Wed, Jun 10, 2009 at 11:49:00AM +0200, andan andan wrote:
> We have two nginx serving static objects from a shared NFS partition,
> and we are using open_file_cache for performance purposes:
>
> With nginx 0.6.x, when the developers uploads files, during a few
> seconds (less than 30s) we see this errors in logs:
>
> [alert] 2721#0: *75303 sendfile() failed (116: Stale NFS file handle)
> while sending response to client, client: XXX ....
>
> I understand that is a normal behaviour because we are using a open_file_cache.
>
> But, when we upgrade to nginx 0.7.59 the same errors appears
> indefinitely (and of course we are sending corrupted objects or
> blank), it seems that, open_file_cache_valid is been ignored in new
> version.
>
> Any idea, tip or recomendation ?
>
> In other hand, is recomended the use of open_file_cache, until now, we
> are very happy with it.
What OS do you using on NFS client and server side ?
How files are uploaded - simple cp or
cp file file.temp
rm file
mv file.temp file
?
I believe in later case NFS should provide atomic operation.
--
Igor Sysoev
http://sysoev.ru/en/
2009/6/10 Igor Sysoev <is at rambler-co.ru>:
> On Wed, Jun 10, 2009 at 11:49:00AM +0200, andan andan wrote:
>
>> We have two nginx serving static objects from a shared NFS partition,
>> and we are using open_file_cache for performance purposes:
>>
>> With nginx 0.6.x, when the developers uploads files, during a few
>> seconds (less than 30s) we see this errors in logs:
>>
>> [alert] 2721#0: *75303 sendfile() failed (116: Stale NFS file handle)
>> while sending response to client, client: XXX ....
>>
>> I understand that is a normal behaviour because we are using a open_file_cache.
>>
>> But, when we upgrade to nginx 0.7.59 the same errors appears
>> indefinitely (and of course we are sending corrupted objects or
>> blank), it seems that, open_file_cache_valid is been ignored in new
>> version.
>>
>> Any idea, tip or recomendation ?
>>
>> In other hand, is recomended the use of open_file_cache, until now, we
>> are very happy with it.
>
> What OS do you using on NFS client and server side ?
Client: RHEL5
2.6.18-128.1.10.el5 x86_64
Server: NAS EMC Celerra NS40
> How files are uploaded - simple cp or
>
> cp file file.temp
> rm file
> mv file.temp file
Normally, they uploads files via scp, but editing a file by hand the
errors appears on the same way.
>
> I believe in later case NFS should provide atomic operation.
Is this related to async/sync mount options ?
Thanks you Igor.
BR.
On Wed, Jun 10, 2009 at 12:24:02PM +0200, andan andan wrote:
> 2009/6/10 Igor Sysoev <is at rambler-co.ru>:
> > On Wed, Jun 10, 2009 at 11:49:00AM +0200, andan andan wrote:
> >
> >> We have two nginx serving static objects from a shared NFS partition,
> >> and we are using open_file_cache for performance purposes:
> >>
> >> With nginx 0.6.x, when the developers uploads files, during a few
> >> seconds (less than 30s) we see this errors in logs:
> >>
> >> [alert] 2721#0: *75303 sendfile() failed (116: Stale NFS file handle)
> >> while sending response to client, client: XXX ....
> >>
> >> I understand that is a normal behaviour because we are using a open_file_cache.
> >>
> >> But, when we upgrade to nginx 0.7.59 the same errors appears
> >> indefinitely (and of course we are sending corrupted objects or
> >> blank), it seems that, open_file_cache_valid is been ignored in new
> >> version.
> >>
> >> Any idea, tip or recomendation ?
> >>
> >> In other hand, is recomended the use of open_file_cache, until now, we
> >> are very happy with it.
> >
> > What OS do you using on NFS client and server side ?
>
> Client: RHEL5
> 2.6.18-128.1.10.el5 x86_64
>
> Server: NAS EMC Celerra NS40
>
> > How files are uploaded - simple cp or
> >
> > cp file file.temp
> > rm file
> > mv file.temp file
>
> Normally, they uploads files via scp, but editing a file by hand the
> errors appears on the same way.
This is because it's no atomic method.
On local filesystem you may get mixed content from old and new versions
without any errors.
> > I believe in later case NFS should provide atomic operation.
>
> Is this related to async/sync mount options ?
No.
--
Igor Sysoev
http://sysoev.ru/en/
2009/6/10 Igor Sysoev <is at rambler-co.ru>:
> On Wed, Jun 10, 2009 at 12:24:02PM +0200, andan andan wrote:
>
>> 2009/6/10 Igor Sysoev <is at rambler-co.ru>:
>> > On Wed, Jun 10, 2009 at 11:49:00AM +0200, andan andan wrote:
>> >
>> >> We have two nginx serving static objects from a shared NFS partition,
>> >> and we are using open_file_cache for performance purposes:
>> >>
>> >> With nginx 0.6.x, when the developers uploads files, during a few
>> >> seconds (less than 30s) we see this errors in logs:
>> >>
>> >> [alert] 2721#0: *75303 sendfile() failed (116: Stale NFS file handle)
>> >> while sending response to client, client: XXX ....
>> >>
>> >> I understand that is a normal behaviour because we are using a open_file_cache.
>> >>
>> >> But, when we upgrade to nginx 0.7.59 the same errors appears
>> >> indefinitely (and of course we are sending corrupted objects or
>> >> blank), it seems that, open_file_cache_valid is been ignored in new
>> >> version.
>> >>
>> >> Any idea, tip or recomendation ?
>> >>
>> >> In other hand, is recomended the use of open_file_cache, until now, we
>> >> are very happy with it.
>> >
>> > What OS do you using on NFS client and server side ?
>>
>> Client: RHEL5
>> 2.6.18-128.1.10.el5 x86_64
>>
>> Server: NAS EMC Celerra NS40
>>
>> > How files are uploaded - simple cp or
>> >
>> > cp file file.temp
>> > rm file
>> > mv file.temp file
>>
>> Normally, they uploads files via scp, but editing a file by hand the
>> errors appears on the same way.
>
> This is because it's no atomic method.
> On local filesystem you may get mixed content from old and new versions
> without any errors.
mmm, then, is recommended to disable open_file_cache in this scenario ?
I'm curious about the question that with nginx 0.6.x in a few seconds
the file is "ready" again, but no with 0.7.x
Thanks again.
On Wed, Jun 10, 2009 at 01:06:32PM +0200, andan andan wrote:
> 2009/6/10 Igor Sysoev <is at rambler-co.ru>:
> > On Wed, Jun 10, 2009 at 12:24:02PM +0200, andan andan wrote:
> >
> >> 2009/6/10 Igor Sysoev <is at rambler-co.ru>:
> >> > On Wed, Jun 10, 2009 at 11:49:00AM +0200, andan andan wrote:
> >> >
> >> >> We have two nginx serving static objects from a shared NFS partition,
> >> >> and we are using open_file_cache for performance purposes:
> >> >>
> >> >> With nginx 0.6.x, when the developers uploads files, during a few
> >> >> seconds (less than 30s) we see this errors in logs:
> >> >>
> >> >> [alert] 2721#0: *75303 sendfile() failed (116: Stale NFS file handle)
> >> >> while sending response to client, client: XXX ....
> >> >>
> >> >> I understand that is a normal behaviour because we are using a open_file_cache.
> >> >>
> >> >> But, when we upgrade to nginx 0.7.59 the same errors appears
> >> >> indefinitely (and of course we are sending corrupted objects or
> >> >> blank), it seems that, open_file_cache_valid is been ignored in new
> >> >> version.
> >> >>
> >> >> Any idea, tip or recomendation ?
> >> >>
> >> >> In other hand, is recomended the use of open_file_cache, until now, we
> >> >> are very happy with it.
> >> >
> >> > What OS do you using on NFS client and server side ?
> >>
> >> Client: RHEL5
> >> 2.6.18-128.1.10.el5 x86_64
> >>
> >> Server: NAS EMC Celerra NS40
> >>
> >> > How files are uploaded - simple cp or
> >> >
> >> > cp file file.temp
> >> > rm file
> >> > mv file.temp file
> >>
> >> Normally, they uploads files via scp, but editing a file by hand the
> >> errors appears on the same way.
> >
> > This is because it's no atomic method.
> > On local filesystem you may get mixed content from old and new versions
> > without any errors.
>
> mmm, then, is recommended to disable open_file_cache in this scenario ?
Even if you will disable open_file_cache, the race condition still remains.
open_file_cache simply makes it distinct: 30s vs 20ms and so.
> I'm curious about the question that with nginx 0.6.x in a few seconds
> the file is "ready" again, but no with 0.7.x
Can not say right now.
--
Igor Sysoev
http://sysoev.ru/en/
2009/6/10 Igor Sysoev <is at rambler-co.ru>:
> On Wed, Jun 10, 2009 at 01:06:32PM +0200, andan andan wrote:
>
>> 2009/6/10 Igor Sysoev <is at rambler-co.ru>:
>> > On Wed, Jun 10, 2009 at 12:24:02PM +0200, andan andan wrote:
>> >
>> >> 2009/6/10 Igor Sysoev <is at rambler-co.ru>:
>> >> > On Wed, Jun 10, 2009 at 11:49:00AM +0200, andan andan wrote:
>> >> >
>> >> >> We have two nginx serving static objects from a shared NFS partition,
>> >> >> and we are using open_file_cache for performance purposes:
>> >> >>
>> >> >> With nginx 0.6.x, when the developers uploads files, during a few
>> >> >> seconds (less than 30s) we see this errors in logs:
>> >> >>
>> >> >> [alert] 2721#0: *75303 sendfile() failed (116: Stale NFS file handle)
>> >> >> while sending response to client, client: XXX ....
>> >> >>
>> >> >> I understand that is a normal behaviour because we are using a open_file_cache.
>> >> >>
>> >> >> But, when we upgrade to nginx 0.7.59 the same errors appears
>> >> >> indefinitely (and of course we are sending corrupted objects or
>> >> >> blank), it seems that, open_file_cache_valid is been ignored in new
>> >> >> version.
>> >> >>
>> >> >> Any idea, tip or recomendation ?
>> >> >>
>> >> >> In other hand, is recomended the use of open_file_cache, until now, we
>> >> >> are very happy with it.
>> >> >
>> >> > What OS do you using on NFS client and server side ?
>> >>
>> >> Client: RHEL5
>> >> 2.6.18-128.1.10.el5 x86_64
>> >>
>> >> Server: NAS EMC Celerra NS40
>> >>
>> >> > How files are uploaded - simple cp or
>> >> >
>> >> > cp file file.temp
>> >> > rm file
>> >> > mv file.temp file
>> >>
>> >> Normally, they uploads files via scp, but editing a file by hand the
>> >> errors appears on the same way.
>> >
>> > This is because it's no atomic method.
>> > On local filesystem you may get mixed content from old and new versions
>> > without any errors.
>>
>> mmm, then, is recommended to disable open_file_cache in this scenario ?
>
> Even if you will disable open_file_cache, the race condition still remains.
> open_file_cache simply makes it distinct: 30s vs 20ms and so.
>
>> I'm curious about the question that with nginx 0.6.x in a few seconds
>> the file is "ready" again, but no with 0.7.x
>
> Can not say right now.
>
Thanks so much for your feedback is very apreciated.
BR.