Migrating our wiki to Confluence

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Migrating our wiki to Confluence

Johan Corveleyn-3
At the hackathon we (me, Stefan Hett, Stefan Fuhrmann, Bert, Julian)
discussed migrating our wiki from the current MoinMoin
(wiki.apache.org/subversion) to the ASF Confluence instance
(cwiki.apache.org). This has been discussed / tried before [1], but
let's have another try.

Motivation:

  * Make it possible to set up pages that anyone can edit, after
creating an account without interaction with some administrator /
moderator (in moinmoin, a new account needs to be activated by someone
with admin privileges -- too much friction).
    - example that triggered this: let's keep a list of distributors /
packagers, and a list of tool vendors / integrators -- would be nice
if people could add themselves to such a page. Apart from that we want
more people to contribute to all kinds of docs / design / discussion.

  * Faster editing / saving (moinmoin still has performance problems).

  * In general: making it easier and nicer for anyone to contribute,
lowering barriers, ...

  * Additional benefit: user accounts are shared between the ASF JIRA
and Confluence instances.


How to do it?
  * Just copy / paste (or export / import) existing pages (keeping the
last author or something), it's no must to keep history.
  * What about existing links to our pages? Can we let them redirect
to the new pages?
  * We should ask around with other ASF projects that have done this.


We still need to figure out some details on the "how", but regardless
we feel we should really just go ahead and do this. I'm willing to
spend some time on this myself (help is very welcome of course), if
the community agrees.

[1] https://svn.haxx.se/dev/archive-2016-10/0171.shtml

--
Johan (& Stefan, Stefan, Bert, Julian)
Reply | Threaded
Open this post in threaded view
|

Re: Migrating our wiki to Confluence

Branko Čibej
On 24.11.2017 12:20, Johan Corveleyn wrote:

> At the hackathon we (me, Stefan Hett, Stefan Fuhrmann, Bert, Julian)
> discussed migrating our wiki from the current MoinMoin
> (wiki.apache.org/subversion) to the ASF Confluence instance
> (cwiki.apache.org). This has been discussed / tried before [1], but
> let's have another try.
>
> Motivation:
>
>   * Make it possible to set up pages that anyone can edit, after
> creating an account without interaction with some administrator /
> moderator (in moinmoin, a new account needs to be activated by someone
> with admin privileges -- too much friction).
>     - example that triggered this: let's keep a list of distributors /
> packagers, and a list of tool vendors / integrators -- would be nice
> if people could add themselves to such a page. Apart from that we want
> more people to contribute to all kinds of docs / design / discussion.
>
>   * Faster editing / saving (moinmoin still has performance problems).
>
>   * In general: making it easier and nicer for anyone to contribute,
> lowering barriers, ...
>
>   * Additional benefit: user accounts are shared between the ASF JIRA
> and Confluence instances.
>
>
> How to do it?
>   * Just copy / paste (or export / import) existing pages (keeping the
> last author or something), it's no must to keep history.
>   * What about existing links to our pages? Can we let them redirect
> to the new pages?
>   * We should ask around with other ASF projects that have done this.
>
>
> We still need to figure out some details on the "how", but regardless
> we feel we should really just go ahead and do this. I'm willing to
> spend some time on this myself (help is very welcome of course), if
> the community agrees.

The most important issues are:
* internal links between pages
* external links to pages

Although if the latter works (with properly configured redirects), the
former should, too.

But if this can't be guaranteed (and automated), you've just volunteered
to do the migration manually. :)

-- Brane
Reply | Threaded
Open this post in threaded view
|

Re: Migrating our wiki to Confluence

Stefan
On 24/11/2017 12:25, Branko Čibej wrote:

> On 24.11.2017 12:20, Johan Corveleyn wrote:
>> At the hackathon we (me, Stefan Hett, Stefan Fuhrmann, Bert, Julian)
>> discussed migrating our wiki from the current MoinMoin
>> (wiki.apache.org/subversion) to the ASF Confluence instance
>> (cwiki.apache.org). This has been discussed / tried before [1], but
>> let's have another try.
>>
>> Motivation:
>>
>>   * Make it possible to set up pages that anyone can edit, after
>> creating an account without interaction with some administrator /
>> moderator (in moinmoin, a new account needs to be activated by someone
>> with admin privileges -- too much friction).
>>     - example that triggered this: let's keep a list of distributors /
>> packagers, and a list of tool vendors / integrators -- would be nice
>> if people could add themselves to such a page. Apart from that we want
>> more people to contribute to all kinds of docs / design / discussion.
>>
>>   * Faster editing / saving (moinmoin still has performance problems).
>>
>>   * In general: making it easier and nicer for anyone to contribute,
>> lowering barriers, ...
>>
>>   * Additional benefit: user accounts are shared between the ASF JIRA
>> and Confluence instances.
>>
>>
>> How to do it?
>>   * Just copy / paste (or export / import) existing pages (keeping the
>> last author or something), it's no must to keep history.
>>   * What about existing links to our pages? Can we let them redirect
>> to the new pages?
>>   * We should ask around with other ASF projects that have done this.
>>
>>
>> We still need to figure out some details on the "how", but regardless
>> we feel we should really just go ahead and do this. I'm willing to
>> spend some time on this myself (help is very welcome of course), if
>> the community agrees.
> The most important issues are:
> * internal links between pages
> * external links to pages
>
> Although if the latter works (with properly configured redirects), the
> former should, too.
>
> But if this can't be guaranteed (and automated), you've just volunteered
> to do the migration manually. :)
>
> -- Brane

The number of pages in the wiki is quite manageable, and I'm more than
willing to help with the migration process.

Regarding redirects/links: Shouldn't be much an issue to simply set
explicit redirects to the URLs one by one from the current wiki pages to
the new locations inside Confluence.

Setting the author of a page needs to be done in the DB directly though,
so we need to check with Infra whether they'd do it for us (I assume atm
we are talking roughly 20-50 pages so it's really not much work to
create an sql-statement for them).

Regards,
Stefan

Reply | Threaded
Open this post in threaded view
|

Re: Migrating our wiki to Confluence

Johan Corveleyn-3
On Fri, Nov 24, 2017 at 1:29 PM, Stefan <[hidden email]> wrote:

> On 24/11/2017 12:25, Branko Čibej wrote:
>> On 24.11.2017 12:20, Johan Corveleyn wrote:
>>> At the hackathon we (me, Stefan Hett, Stefan Fuhrmann, Bert, Julian)
>>> discussed migrating our wiki from the current MoinMoin
>>> (wiki.apache.org/subversion) to the ASF Confluence instance
>>> (cwiki.apache.org). This has been discussed / tried before [1], but
>>> let's have another try.
>>>
>>> Motivation:
>>>
>>>   * Make it possible to set up pages that anyone can edit, after
>>> creating an account without interaction with some administrator /
>>> moderator (in moinmoin, a new account needs to be activated by someone
>>> with admin privileges -- too much friction).
>>>     - example that triggered this: let's keep a list of distributors /
>>> packagers, and a list of tool vendors / integrators -- would be nice
>>> if people could add themselves to such a page. Apart from that we want
>>> more people to contribute to all kinds of docs / design / discussion.
>>>
>>>   * Faster editing / saving (moinmoin still has performance problems).
>>>
>>>   * In general: making it easier and nicer for anyone to contribute,
>>> lowering barriers, ...
>>>
>>>   * Additional benefit: user accounts are shared between the ASF JIRA
>>> and Confluence instances.
>>>
>>>
>>> How to do it?
>>>   * Just copy / paste (or export / import) existing pages (keeping the
>>> last author or something), it's no must to keep history.
>>>   * What about existing links to our pages? Can we let them redirect
>>> to the new pages?
>>>   * We should ask around with other ASF projects that have done this.
>>>
>>>
>>> We still need to figure out some details on the "how", but regardless
>>> we feel we should really just go ahead and do this. I'm willing to
>>> spend some time on this myself (help is very welcome of course), if
>>> the community agrees.
>> The most important issues are:
>> * internal links between pages
>> * external links to pages
>>
>> Although if the latter works (with properly configured redirects), the
>> former should, too.
>>
>> But if this can't be guaranteed (and automated), you've just volunteered
>> to do the migration manually. :)
>>
>> -- Brane
>
> The number of pages in the wiki is quite manageable, and I'm more than
> willing to help with the migration process.
>
> Regarding redirects/links: Shouldn't be much an issue to simply set
> explicit redirects to the URLs one by one from the current wiki pages to
> the new locations inside Confluence.
>
> Setting the author of a page needs to be done in the DB directly though,
> so we need to check with Infra whether they'd do it for us (I assume atm
> we are talking roughly 20-50 pages so it's really not much work to
> create an sql-statement for them).

I've tried to get a list of all our current wiki pages. This search
gives 105 pages, but some can be ignored (some system pages, and the
"profile pages" of each user):

https://wiki.apache.org/subversion/FindPage?advancedsearch=1&and_terms=.&mtime=8+years+before&action=fullsearch

Instead of using this search (with mtime=8+years+before or somesuch,
i.e. fill in '8 years before', and a '.' for search terms, in
https://wiki.apache.org/subversion/FindPage), one can also list all
pages (https://wiki.apache.org/subversion/TitleIndex), but that gives
many more irrelevant pages, as it also includes help pages from
MoinMoin itself.

Since I like history (I'm interested in version control after all) I'd
like to try to import them into Confluence with history. There seems
to be a tool to export pages (optionally with history) from MoinMoin,
to convert them into something that can be fed to the "Universal Wiki
Converter" [1]. I'll read a bit more about that, and interface with
infra to see if we can give that a try.

Also, two INFRA jira tickets I found interesting:
* https://issues.apache.org/jira/browse/INFRA-7256 (Create Confluence
wiki for CouchDB) -- this one had a reference to the Wiki Converter
* https://issues.apache.org/jira/browse/INFRA-10942 (Moving Cassandra
wiki to confluence) -- this one mentioned "As for 302 the moin wiki
instance has the ability for you to redirect pages themselves", as
Stefan also suggested above.


[1] https://migrations.atlassian.net/wiki/spaces/UWC/pages/1015848/UWC+MoinMoin+Notes

--
Johan
Reply | Threaded
Open this post in threaded view
|

Re: Migrating our wiki to Confluence

Julian Foad-5
Johan Corveleyn wrote:
> I've tried to get a list of all our current wiki pages. [...]

Thanks for the progress update.

I notice there are several "orphaned" pages that are useful. For
example, "Berlin2015" was orphaned; I have just linked it from "Meetings".

> Since I like history (I'm interested in version control after all) I'd
> like to try to import them into Confluence with history. There seems
> to be a tool to export pages (optionally with history) from MoinMoin,
> to convert them into something that can be fed to the "Universal Wiki
> Converter" [1]. I'll read a bit more about that, and interface with
> infra to see if we can give that a try.

That would feel good because we like versioned history, although I think
it is not important so don't hold up the progress because of it.

- Julian


> Also, two INFRA jira tickets I found interesting:
> * https://issues.apache.org/jira/browse/INFRA-7256 (Create Confluence
> wiki for CouchDB) -- this one had a reference to the Wiki Converter
> * https://issues.apache.org/jira/browse/INFRA-10942 (Moving Cassandra
> wiki to confluence) -- this one mentioned "As for 302 the moin wiki
> instance has the ability for you to redirect pages themselves", as
> Stefan also suggested above.
>
>
> [1] https://migrations.atlassian.net/wiki/spaces/UWC/pages/1015848/UWC+MoinMoin+Notes

Reply | Threaded
Open this post in threaded view
|

Re: Migrating our wiki to Confluence

Stefan Hett-2
In reply to this post by Johan Corveleyn-3
On 12/1/2017 5:07 PM, Johan Corveleyn wrote:

> On Fri, Nov 24, 2017 at 1:29 PM, Stefan <[hidden email]> wrote:
>> On 24/11/2017 12:25, Branko Čibej wrote:
>>> On 24.11.2017 12:20, Johan Corveleyn wrote:
>>>> At the hackathon we (me, Stefan Hett, Stefan Fuhrmann, Bert, Julian)
>>>> discussed migrating our wiki from the current MoinMoin
>>>> (wiki.apache.org/subversion) to the ASF Confluence instance
>>>> (cwiki.apache.org). This has been discussed / tried before [1], but
>>>> let's have another try.
>>>>
>>>> Motivation:
>>>>
>>>>    * Make it possible to set up pages that anyone can edit, after
>>>> creating an account without interaction with some administrator /
>>>> moderator (in moinmoin, a new account needs to be activated by someone
>>>> with admin privileges -- too much friction).
>>>>      - example that triggered this: let's keep a list of distributors /
>>>> packagers, and a list of tool vendors / integrators -- would be nice
>>>> if people could add themselves to such a page. Apart from that we want
>>>> more people to contribute to all kinds of docs / design / discussion.
>>>>
>>>>    * Faster editing / saving (moinmoin still has performance problems).
>>>>
>>>>    * In general: making it easier and nicer for anyone to contribute,
>>>> lowering barriers, ...
>>>>
>>>>    * Additional benefit: user accounts are shared between the ASF JIRA
>>>> and Confluence instances.
>>>>
>>>>
>>>> How to do it?
>>>>    * Just copy / paste (or export / import) existing pages (keeping the
>>>> last author or something), it's no must to keep history.
>>>>    * What about existing links to our pages? Can we let them redirect
>>>> to the new pages?
>>>>    * We should ask around with other ASF projects that have done this.
>>>>
>>>>
>>>> We still need to figure out some details on the "how", but regardless
>>>> we feel we should really just go ahead and do this. I'm willing to
>>>> spend some time on this myself (help is very welcome of course), if
>>>> the community agrees.
>>> The most important issues are:
>>> * internal links between pages
>>> * external links to pages
>>>
>>> Although if the latter works (with properly configured redirects), the
>>> former should, too.
>>>
>>> But if this can't be guaranteed (and automated), you've just volunteered
>>> to do the migration manually. :)
>>>
>>> -- Brane
>> The number of pages in the wiki is quite manageable, and I'm more than
>> willing to help with the migration process.
>>
>> Regarding redirects/links: Shouldn't be much an issue to simply set
>> explicit redirects to the URLs one by one from the current wiki pages to
>> the new locations inside Confluence.
>>
>> Setting the author of a page needs to be done in the DB directly though,
>> so we need to check with Infra whether they'd do it for us (I assume atm
>> we are talking roughly 20-50 pages so it's really not much work to
>> create an sql-statement for them).
> I've tried to get a list of all our current wiki pages. This search
> gives 105 pages, but some can be ignored (some system pages, and the
> "profile pages" of each user):
>
> https://wiki.apache.org/subversion/FindPage?advancedsearch=1&and_terms=.&mtime=8+years+before&action=fullsearch
>
> Instead of using this search (with mtime=8+years+before or somesuch,
> i.e. fill in '8 years before', and a '.' for search terms, in
> https://wiki.apache.org/subversion/FindPage), one can also list all
> pages (https://wiki.apache.org/subversion/TitleIndex), but that gives
> many more irrelevant pages, as it also includes help pages from
> MoinMoin itself.
Correct, https://wiki.apache.org/subversion/TitleIndex is the list I
skimmed through when estimating the number of to be migrated pages.
Excluding all the moin-wiki-help pages and the ones which seem to not
work at all (i.e. the Chinese ones for example just give me internal
server errors) you'd come up with a rough estimate of around 50 pages or
so. Excluding pages which should presumably be filtered out too (f.e.
https://wiki.apache.org/subversion/OsieroconeStrony) reduces the amount
a bit further.

>
> Since I like history (I'm interested in version control after all) I'd
> like to try to import them into Confluence with history. There seems
> to be a tool to export pages (optionally with history) from MoinMoin,
> to convert them into something that can be fed to the "Universal Wiki
> Converter" [1]. I'll read a bit more about that, and interface with
> infra to see if we can give that a try.
>
> Also, two INFRA jira tickets I found interesting:
> * https://issues.apache.org/jira/browse/INFRA-7256 (Create Confluence
> wiki for CouchDB) -- this one had a reference to the Wiki Converter
> * https://issues.apache.org/jira/browse/INFRA-10942 (Moving Cassandra
> wiki to confluence) -- this one mentioned "As for 302 the moin wiki
> instance has the ability for you to redirect pages themselves", as
> Stefan also suggested above.
>
>
> [1] https://migrations.atlassian.net/wiki/spaces/UWC/pages/1015848/UWC+MoinMoin+Notes
>
UWC is what's a usable way to migrate wikis to Conflunece. But the last
time I reviewed this (which was around the time of the Confluence 4.x
releases as far as I remember) it suggested it's some work and not an
all-in-one solution which can easily be applied. Ultimately this is why
we in our company ended up migrating things manually. If you wanna give
it a try, spend the required time to work out the details, and Infra
providing the (presumably) required support to import the data, I'm all
for going that way instead of simply moving all pages over manually. In
the end it has been a couple of years since I looked at UWC and things
might be easier to achieve now than they were back then.

That said, thanks for picking this up, Johan.

--
Regards,
Stefan Hett

Reply | Threaded
Open this post in threaded view
|

Re: Migrating our wiki to Confluence

Johan Corveleyn-3
In reply to this post by Johan Corveleyn-3
On Fri, Nov 24, 2017 at 12:20 PM, Johan Corveleyn <[hidden email]> wrote:
...

> Motivation:
>
>   * Make it possible to set up pages that anyone can edit, after
> creating an account without interaction with some administrator /
> moderator (in moinmoin, a new account needs to be activated by someone
> with admin privileges -- too much friction).
>     - example that triggered this: let's keep a list of distributors /
> packagers, and a list of tool vendors / integrators -- would be nice
> if people could add themselves to such a page. Apart from that we want
> more people to contribute to all kinds of docs / design / discussion.

After chatting on #asfinfra (hipchat) with the experts, it seems the
above is not an option. The "self-signup" is not well protected (no
captcha), so creating pages with liberal permissions (anyone with an
account can edit) is a recipe for spam. Maybe we can come up with a
simple, pleasant workflow for new people to get added to an
editor-group on demand, but that's about it.

>   * Faster editing / saving (moinmoin still has performance problems).
>
>   * In general: making it easier and nicer for anyone to contribute,
> lowering barriers, ...
>
>   * Additional benefit: user accounts are shared between the ASF JIRA
> and Confluence instances.

And shared accounts are not a reality at this time either. Though
infra are going to make this happen sometime next year (by installing
"crowd", specific authentication service of Atlassian, enabling SSO
accross their services coupled with it).

Although the more liberal editing is not an option, I still think we
should go forward with migrating to Confluence. For the other reasons
(more attractive to all kinds of contributors (also non-coders), lower
barriers, faster editing/saving, ...). Plus: in the longer term, infra
wants to end-of-life their moinmoin service, if I understood
correctly.

gmcdonald will be sending me the Universal Wiki Converter tools and
actual moinmoin pages of Subversion, so I can experiment a bit with
it.

--
Johan
Reply | Threaded
Open this post in threaded view
|

Re: Migrating our wiki to Confluence

Stefan
On 02/12/2017 19:55, Johan Corveleyn wrote:

> On Fri, Nov 24, 2017 at 12:20 PM, Johan Corveleyn <[hidden email]> wrote:
> ...
>> Motivation:
>>
>>   * Make it possible to set up pages that anyone can edit, after
>> creating an account without interaction with some administrator /
>> moderator (in moinmoin, a new account needs to be activated by someone
>> with admin privileges -- too much friction).
>>     - example that triggered this: let's keep a list of distributors /
>> packagers, and a list of tool vendors / integrators -- would be nice
>> if people could add themselves to such a page. Apart from that we want
>> more people to contribute to all kinds of docs / design / discussion.
> After chatting on #asfinfra (hipchat) with the experts, it seems the
> above is not an option. The "self-signup" is not well protected (no
> captcha), so creating pages with liberal permissions (anyone with an
> account can edit) is a recipe for spam. Maybe we can come up with a
> simple, pleasant workflow for new people to get added to an
> editor-group on demand, but that's about it.
Maybe this is an argument for prioritizing something on my side which
atm is WiP. It could help with the situation if it's interesting for
INFRA, but it won't be ready within the next couple of weeks (i.e.
thinking a time frame around the first half next year). Sry, can't say
more publicly about this just yet, but it might be exactly the thing
which could allow INFRA to reconsider their decision on the self-signup
restrictions.

>
>>   * Faster editing / saving (moinmoin still has performance problems).
>>
>>   * In general: making it easier and nicer for anyone to contribute,
>> lowering barriers, ...
>>
>>   * Additional benefit: user accounts are shared between the ASF JIRA
>> and Confluence instances.
> And shared accounts are not a reality at this time either. Though
> infra are going to make this happen sometime next year (by installing
> "crowd", specific authentication service of Atlassian, enabling SSO
> accross their services coupled with it).
Good to know that they are working on a proper integration. So if that's
currently scheduled to happen next year, that'd be fine for us, I take it.
> Although the more liberal editing is not an option, I still think we
> should go forward with migrating to Confluence. For the other reasons
> (more attractive to all kinds of contributors (also non-coders), lower
> barriers, faster editing/saving, ...). Plus: in the longer term, infra
> wants to end-of-life their moinmoin service, if I understood
> correctly.
I'm still all for doing the migration, regardless of the pointed out
restrictions.
> gmcdonald will be sending me the Universal Wiki Converter tools and
> actual moinmoin pages of Subversion, so I can experiment a bit with
> it.
>
Great. Hope there's not too much work required there.

Regards,
Stefan

Reply | Threaded
Open this post in threaded view
|

Re: Migrating our wiki to Confluence

Johan Corveleyn-3
On Sat, Dec 2, 2017 at 10:34 PM, Stefan <[hidden email]> wrote:

> On 02/12/2017 19:55, Johan Corveleyn wrote:
>> On Fri, Nov 24, 2017 at 12:20 PM, Johan Corveleyn <[hidden email]> wrote:
>> ...
>>> Motivation:
>>>
>>>   * Make it possible to set up pages that anyone can edit, after
>>> creating an account without interaction with some administrator /
>>> moderator (in moinmoin, a new account needs to be activated by someone
>>> with admin privileges -- too much friction).
>>>     - example that triggered this: let's keep a list of distributors /
>>> packagers, and a list of tool vendors / integrators -- would be nice
>>> if people could add themselves to such a page. Apart from that we want
>>> more people to contribute to all kinds of docs / design / discussion.
>> After chatting on #asfinfra (hipchat) with the experts, it seems the
>> above is not an option. The "self-signup" is not well protected (no
>> captcha), so creating pages with liberal permissions (anyone with an
>> account can edit) is a recipe for spam. Maybe we can come up with a
>> simple, pleasant workflow for new people to get added to an
>> editor-group on demand, but that's about it.
> Maybe this is an argument for prioritizing something on my side which
> atm is WiP. It could help with the situation if it's interesting for
> INFRA, but it won't be ready within the next couple of weeks (i.e.
> thinking a time frame around the first half next year). Sry, can't say
> more publicly about this just yet, but it might be exactly the thing
> which could allow INFRA to reconsider their decision on the self-signup
> restrictions.

Cool. Looking forward to it :-).

Just to clarify: infra doesn't restrict self-signup (they'd like to
have a good captcha system, but apparently the one from Atlassian
doesn't work well), but they do advise strongly against setting
permissions on pages that allow editing by any authenticated user.
Apparently that ends up with spam sooner or later.

--
Johan